diff --git a/README.md b/README.md index 2aad5ee89..cfaf59cce 100644 --- a/README.md +++ b/README.md @@ -249,6 +249,8 @@ One-time supporters can ask to be listed too. MIT, © 2026 Myeonggu Kim (KnifeLemon) and Filee contributors. Bundled third-party components keep their own licenses, listed in [THIRD-PARTY-NOTICES.md](THIRD-PARTY-NOTICES.md). +본 제품은 한글과컴퓨터의 한/글 문서 파일(.hwp) 공개 문서를 참고하여 개발하였습니다. (Filee's HWP and HWPX code was developed with reference to Hancom's published HWP file format document.) +

Star history

diff --git a/THIRD-PARTY-NOTICES.md b/THIRD-PARTY-NOTICES.md index 59e8e8d95..dc7c7e8f0 100644 --- a/THIRD-PARTY-NOTICES.md +++ b/THIRD-PARTY-NOTICES.md @@ -108,6 +108,15 @@ Windows headers, without a C runtime. Pieces of these projects that can end up i the x86 branch filter of the LZMA SDK (public domain, Igor Pavlov, https://www.7-zip.org/sdk.html), and the published app as its payload. Filee 1.x's installer was built with Inno Setup, which is no longer used. +## Hancom HWP document format + +Filee's own HWP and HWPX code (the layout engine and the readers it uses) was developed with reference to Hancom's +published document on the HWP file format, 「한글 문서 파일 형식 5.0」, which asks for this notice: + +본 제품은 한글과컴퓨터의 한/글 문서 파일(.hwp) 공개 문서를 참고하여 개발하였습니다. + +(This product was developed with reference to Hancom's published document on the Hangul (.hwp) file format.) + ## Unicode data `src/Filee.Engines/Typesetting/UnicodeData.Generated.cs` holds Line_Break, Bidi_Class, Script and East_Asian_Width diff --git a/build/render-compare.ps1 b/build/render-compare.ps1 index 86df7f8cf..42f3b568f 100644 --- a/build/render-compare.ps1 +++ b/build/render-compare.ps1 @@ -43,7 +43,7 @@ $project = Join-Path $repository 'tests/Filee.Engines.Tests/Filee.Engines.Tests. $env:FILEE_RENDER_CORPUS = (Resolve-Path $Corpus).Path $env:FILEE_RENDER_REPORT = [IO.Path]::GetFullPath($Report) try { - dotnet test $project -c $Configuration --filter 'FullyQualifiedName~RenderQuality.RenderCorpusTests' --logger 'console;verbosity=normal' + dotnet test $project -c $Configuration --filter 'FullyQualifiedName~RenderQuality.RenderCorpusTests|FullyQualifiedName~RenderQuality.HangulLineBreakTests' --logger 'console;verbosity=normal' if ($LASTEXITCODE -ne 0) { throw "The compare failed (exit $LASTEXITCODE)." } } finally { diff --git a/docs/ENGINES.md b/docs/ENGINES.md index 1e85c7a18..116db9205 100644 --- a/docs/ENGINES.md +++ b/docs/ENGINES.md @@ -720,6 +720,66 @@ properties) with every installed font of the script, against recorded HarfBuzz o font (joining forms and lam-alef, vowel signs drawn before their consonant, conjuncts, the SARA AM split, jamo composed into precomposed syllables, a dotted circle only for a broken cluster). +## Filee renderer: line breaking and paragraph layout + +본 제품은 한글과컴퓨터의 한/글 문서 파일(.hwp) 공개 문서를 참고하여 개발하였습니다. (The layout engine's HWP and HWPX handling was checked against Hancom's published HWP 5.0 file format +document, whose license asks for this notice in the product's interface, manuals, help and source; it is also in +THIRD-PARTY-NOTICES.md, which the About page shows.) + +Paragraph layout of Filee's own renderer (GitHub Project 4, item *Layout engine: paragraph layout and line breaking*), +in `src/Filee.Engines/Typesetting`. No converter uses it yet; the render quality harness measures it against rhwp. + +- **Line breaking** (`LineBreaker.cs`): UAX #14 at Unicode 16.0, rules LB1–LB31; it passes all 16,672 cases of + Unicode's `LineBreakTest.txt` (checked when written; the file isn't kept, `LineBreakerTests` covers each rule + family). Tailorings: 한글 어절 keeps Hangul words whole, 글자 breaks between syllables as UAX #14 does (한글 writes + them the other way round from the names: `breakNonLatinWord` BREAK_WORD is 어절, KEEP_WORD 글자; the binary HWP + of the same document and 한글's own line layout say so); Latin words broken anywhere (`breakLatinWord` BREAK_WORD, + Word without `w:wordWrap`); a document's own characters forbidden at a line's start or end (Word + `w:noLineBreaksBefore` / `After`). Hyphenation keeps words whole (no dictionaries), and Thai, Lao, Khmer and Myanmar + break only at spaces until there is dictionary breaking. +- **Paragraph text** (`ParagraphText.cs`): the inlines as one string (tabs, line breaks as U+2028, links and bookmarks + kept, a list item's number or bullet in front, followed by a tab when the paragraph hangs) and each character's style: + the font, 장평, 자간, relative size and 글자 위치 of its 한글 language slot, or Word's run font, scale, absolute + spacing and raise. 한글's 자간 is a share of each character's own width (장평 applied), not of the font size, so a + space or a narrowed character closes up less (한글's stored line layout breaks where this puts the line end). 한글's + "글꼴에 어울리는 빈칸" off (`useFontSpace="0"`, as in Filee's template) makes a space half the size + wide, and `useKerning="0"` turns the font's kerning off; both are read into `HCharLayout`. +- **Layout** (`ParagraphLayout.cs`, output in `LineBox.cs`): items of one style, font, script and direction are shaped + (`Shaper`), lines fill greedily to the column width and break at the last opportunity (a word wider than the line + between clusters), spaces hang at the line end. 최소 공백 (`condense`) keeps a word that fits once the spaces + shrink; `lineWrap` KEEP and SQUEEZE hold the paragraph on one line. Each line's parts are shaped again with their + context, so a break inside an Arabic word or after an Indic cluster shapes as it ends. + - Alignment: left, right, centre; justify widens the spaces (the clusters where a line has none) on all but the last + line; distribute (배분) widens between all clusters, the last line too. Indents and hanging indents as the model + has them; the start indent of a right-to-left paragraph is on the right. + - Tabs: left, right, centre and decimal stops with dot, dash and solid leaders, measured from the paragraph's left + indent as 한글 does; default stops every 40 pt; a hanging indent is a stop on the first line. + - Line height by the paragraph's rule: 한글's percent, fixed, at least and between lines measure from the font size + with the baseline at 85% of the text height (as 한글's stored line segments put it); Word's multiple and 한글's + "글꼴에 어울리는 줄 높이" from the fonts' OS/2 win ascent and descent. + - Characters: size, ratio, letter spacing, baseline shift, super- and subscript, colour, shade, underline, + strike-through, links and bookmarks; synthetic bold and italic (`GlyphRun.Embolden` / `SkewX`) for faces a family + lacks. + - Direction: a simplified bidirectional algorithm (strong letters and numbers set the levels, neutrals take the + direction around them, rule L2 reorders). The runs stay in logical order in the page, so the PDF's text reads as + it was written. Explicit embeddings, isolates and brackets come with the bidi milestone. +- **Interim page flow** (`FileeRenderer.cs`): each section's pages with its paper and margins, paragraphs stacked down + the body with a new page where the next line doesn't fit, headings as the PDF outline. It stands in for the page + composer; table cells are stacked as paragraphs and objects left out until then. + +The PDF writer's subsets now keep only the characters the text has in their cmap: CJK fonts map 文 and the Kangxi +radical ⽂ to one glyph, and the PDF's ToUnicode took the lower code point, so Japanese and Chinese text copied out as +radicals. + +Tests: `LineBreakerTests`, `ParagraphLayoutTests` (with the pinned Noto fonts of `build/fetch-fonts.ps1`), and the +render quality harness, where the renderer is `DocumentRenderers.Candidate`. Against rhwp, Japanese, Chinese, lists +and tabs break into the same lines on the same baselines; rhwp splits some Korean words even in 어절 paragraphs. The +rest waits for the page composer. Against 한글 itself, `HangulLineBreakTests` compares the line starts 한글 stored in +its HWPX files (`hp:linesegarray`) with Filee's, for the plain-text paragraphs of a private corpus +(`FILEE_RENDER_CORPUS`): on the HWPX files attached to Filee's issues, 150 of 152 paragraphs break where 한글 broke +them. Of the other two, one uses a font that isn't installed, and the other is a run of 75 hyphens wider than the +line, which 한글 keeps on one line. + ## DOCX writer The counterpart of the HWPX writer: `Hwp/Hwpx/Docx/DocxWriter*.cs` writes WordprocessingML by hand from the same diff --git a/docs/RENDER-QUALITY.md b/docs/RENDER-QUALITY.md index 6be05dfc6..e50401111 100644 --- a/docs/RENDER-QUALITY.md +++ b/docs/RENDER-QUALITY.md @@ -22,9 +22,9 @@ internal sealed record DocumentRenderer(string Name, IReadOnlySet Format (`RhwpInput`'s fixes, then `rhwp export-pdf` with Filee's fallback fonts). rhwp is only the yardstick here; nothing in the harness adds a conversion path through it. -`DocumentRenderers.Candidate` is Filee's layout engine. It is null until the first layout milestone sets it, and until -then the compare test is skipped with that reason. Once it is set, its PDFs get the same read-back checks and are -compared with rhwp's, with no other change. +`DocumentRenderers.Candidate` is Filee's layout engine: paragraph layout on an interim page flow +(`Typesetting/FileeRenderer.cs`) until the page composer replaces it. Its PDFs get the same read-back checks and are +compared with rhwp's; set to null, the compare test is skipped. ## The feature matrix @@ -111,6 +111,9 @@ shaping. holds documents next to the PDFs 한글 printed of them (`doc.hwpx` + `doc.pdf`, `doc.hwp` + `doc.pdf`). Each renderer is compared with 한글's PDF and read back. An HTML report goes to the temp folder, with the page that differs most of each document side by side. The corpus run only reports; it never fails on the numbers. + `HangulLineBreakTests` runs on the same folder: for every HWPX 한글 saved (no PDF needed), the plain-text + paragraphs are laid out by Filee at the width 한글 used, and their line starts are compared with the ones 한글 + stored (`hp:linesegarray`). ## rhwp today diff --git a/src/Filee.App/Assets/i18n/id.json b/src/Filee.App/Assets/i18n/id.json index 8d602a6cb..89b015e81 100644 --- a/src/Filee.App/Assets/i18n/id.json +++ b/src/Filee.App/Assets/i18n/id.json @@ -132,6 +132,7 @@ "error.fon_unsupported": "Jenis font .fon ini tidak didukung", "error.fon_too_large": "Font .fon terlalu besar untuk dikonversi", "error.fon_damaged": "File .fon rusak atau terpotong", + "error.document_too_complex": "Dokumen terlalu besar atau terlalu rumit untuk dibaca dengan aman", "error.m4s_needs_init": "File .m4s ini adalah salah satu pecahan video streaming: tidak dapat dibaca tanpa segmen awal (init) terpisahnya", "notice.pdf_not_smaller": "Ukuran tidak dapat diperkecil lagi, sehingga disalin apa adanya", "notice.under_min_size": "Disimpan, tetapi lebih kecil dari ukuran file minimum", diff --git a/src/Filee.App/Assets/i18n/it.json b/src/Filee.App/Assets/i18n/it.json index ed2bf8708..69bfbdb05 100644 --- a/src/Filee.App/Assets/i18n/it.json +++ b/src/Filee.App/Assets/i18n/it.json @@ -132,6 +132,7 @@ "error.fon_unsupported": "Questo tipo di font .fon non è supportato", "error.fon_too_large": "Il font .fon è troppo grande per essere convertito", "error.fon_damaged": "Il file .fon è danneggiato o troncato", +"error.document_too_complex": "Il documento è troppo grande o troppo complesso per essere letto in sicurezza", "error.m4s_needs_init": "Questo .m4s è un frammento di un video in streaming: non può essere letto senza il suo segmento iniziale (init) separato", "notice.pdf_not_smaller": "Impossibile ridurlo, quindi è stato copiato così com'è", "notice.under_min_size": "Salvato, ma più piccolo della dimensione minima del file", diff --git a/src/Filee.Engines/Fonts/FontSubsetter.cs b/src/Filee.Engines/Fonts/FontSubsetter.cs index c9d853729..8b48375d3 100644 --- a/src/Filee.Engines/Fonts/FontSubsetter.cs +++ b/src/Filee.Engines/Fonts/FontSubsetter.cs @@ -55,9 +55,14 @@ public static bool CanSubset(SfntFont font, out string? reason) /// A TrueType font with the outlines of (plus .notdef and the components they use) at /// their original glyph IDs and every other glyph empty. /// + /// + /// The characters drawn with the font, or null: the cmap then keeps only these, so a glyph several characters share + /// (文 and the Kangxi radical ⽂ in CJK fonts) maps back to the character the text has, not the lowest one. + /// /// The font is one refuses. /// A glyph ID is outside the font. - public static SfntFont Subset(SfntFont font, IEnumerable glyphs, CancellationToken cancellationToken = default) + public static SfntFont Subset(SfntFont font, IEnumerable glyphs, CancellationToken cancellationToken = default, + IReadOnlySet? codePoints = null) { if (!CanSubset(font, out var reason)) throw new NotSupportedException(reason); @@ -95,7 +100,8 @@ public static SfntFont Subset(SfntFont font, IEnumerable glyphs, Cancellati if (source["cmap"] is { } cmap) { var map = CmapTable.Read(cmap, out var symbol); - subset.Tables["cmap"] = CmapTable.Build(map.Where(p => used.Contains(p.Value)).ToDictionary(), symbol); + subset.Tables["cmap"] = CmapTable.Build( + map.Where(p => used.Contains(p.Value) && (codePoints is null || codePoints.Contains(p.Key))).ToDictionary(), symbol); } else { diff --git a/src/Filee.Engines/Hwp/Hwpx/HwpxModel.Layout.cs b/src/Filee.Engines/Hwp/Hwpx/HwpxModel.Layout.cs index 7f45d5b12..8e9fdb73f 100644 --- a/src/Filee.Engines/Hwp/Hwpx/HwpxModel.Layout.cs +++ b/src/Filee.Engines/Hwp/Hwpx/HwpxModel.Layout.cs @@ -15,12 +15,19 @@ namespace Filee.Engines.Hwp.Hwpx; /// Word character scale (w:w) in percent: the width of every glyph, like 한글's 장평. /// Word letter spacing (w:spacing) as an absolute distance added after each character. /// Word baseline shift (w:position): positive raises the text. +/// +/// 한글 "글꼴에 어울리는 빈칸" (useFontSpace): a space is as wide as the font's own; off, it is half the font size. +/// Null where the source has no such setting (the font's own width). +/// +/// 한글 "커닝" (useKerning): the font's kerning applies; null = it does (as in Word). internal sealed record HCharLayout( HLanguageSlots? Slots = null, HWordFonts? WordFonts = null, int? Scale = null, int? LetterSpacing = null, - int? Position = null) + int? Position = null, + bool? FontSpace = null, + bool? Kerning = null) { /// This layout with the members it leaves open taken from . public HCharLayout Over(HCharLayout? baseline) => baseline is null ? this : new HCharLayout( @@ -28,7 +35,9 @@ internal sealed record HCharLayout( WordFonts ?? baseline.WordFonts, Scale ?? baseline.Scale, LetterSpacing ?? baseline.LetterSpacing, - Position ?? baseline.Position); + Position ?? baseline.Position, + FontSpace ?? baseline.FontSpace, + Kerning ?? baseline.Kerning); } /// 한글's seven language slots (글꼴 언어): each character takes the properties of the slot its script falls into. @@ -176,7 +185,10 @@ internal sealed record HParaBorder(HBorders Borders, string? Fill, HInsets Offse /// Where lines may break and how they are fitted. Null members are left to the style (or the default rules). /// Latin words: kept whole (어절), hyphenated or broken anywhere (한글 breakLatinWord, Word w:wordWrap). -/// Korean words: kept whole (어절) or broken at any syllable (글자) (한글 breakNonLatinWord). +/// +/// Korean words: kept whole (어절) or broken at any syllable (글자) (한글 breakNonLatinWord, whose BREAK_WORD is 어절 and +/// KEEP_WORD 글자). +/// /// 한글 lineWrap: wrap, squeeze the line, or keep it on one line. /// East Asian line start / end rules apply (Word w:kinsoku; 한글 applies its 금칙 characters). /// Punctuation may hang past the line end (Word w:overflowPunct). diff --git a/src/Filee.Engines/Hwp/Hwpx/HwpxReader.Header.cs b/src/Filee.Engines/Hwp/Hwpx/HwpxReader.Header.cs index 1902ffbae..02ee10533 100644 --- a/src/Filee.Engines/Hwp/Hwpx/HwpxReader.Header.cs +++ b/src/Filee.Engines/Hwp/Hwpx/HwpxReader.Header.cs @@ -98,7 +98,7 @@ private HCharFormat CharFormat(string? id) Color: Color(Attr(charPr, "textColor")) ?? "#000000", Shade: Color(Attr(charPr, "shadeColor")), Spacing: Int(Child(charPr, "spacing"), "hangul"), - Layout: new HCharLayout(Slots(charPr, Face))); + Layout: new HCharLayout(Slots(charPr, Face), FontSpace: Flag(charPr, "useFontSpace"), Kerning: Flag(charPr, "useKerning"))); return _charFormats[id] = format; } @@ -175,7 +175,12 @@ private HParaLayout ParaLayout(XElement paraPr, XElement? breaks, HLineHeight li "BREAK_WORD" => HLatinBreak.BreakWord, _ => HLatinBreak.KeepWord, }, - NonLatin: Attr(breaks, "breakNonLatinWord") == "BREAK_WORD" ? HNonLatinBreak.BreakWord : HNonLatinBreak.KeepWord, + // 한글 writes the 한글 단위 setting the other way round from its names: KEEP_WORD for 글자 (a break at any + // syllable) and BREAK_WORD for 어절. The binary HWP of the same document has bit 7 of the paragraph shape set + // (1 = 글자 in Hancom's HWP 5.0 format document) where the HWPX says KEEP_WORD, and 한글's own line layout of + // KEEP_WORD paragraphs breaks inside words. + // 본 제품은 한글과컴퓨터의 한/글 문서 파일(.hwp) 공개 문서를 참고하여 개발하였습니다. + NonLatin: Attr(breaks, "breakNonLatinWord") == "BREAK_WORD" ? HNonLatinBreak.KeepWord : HNonLatinBreak.BreakWord, LineWrap: Attr(breaks, "lineWrap") switch { "SQUEEZE" => HLineWrap.Squeeze, diff --git a/src/Filee.Engines/Rendering/PageContent.cs b/src/Filee.Engines/Rendering/PageContent.cs index 10529cb05..f868660da 100644 --- a/src/Filee.Engines/Rendering/PageContent.cs +++ b/src/Filee.Engines/Rendering/PageContent.cs @@ -21,6 +21,15 @@ internal sealed record GlyphRun(PageFont Font, float Size, ushort[] Glyphs, SKPo /// Horizontal scale of the glyphs (1 = as designed; 한글's 장평). public float ScaleX { get; init; } = 1; + /// Bold the font lacks, drawn by thickening the outlines. + public bool Embolden { get; init; } + + /// Horizontal skew of the glyphs (italic the font lacks: ), 0 = upright. + public float SkewX { get; init; } + + /// The skew of synthetic italic, as Skia and browsers slant (about 12°). + public const float SyntheticSlant = -0.2f; + /// Throws when the arrays don't describe the same glyphs or a cluster points outside the text. public void Validate() { @@ -119,6 +128,8 @@ private static void DrawRun(SKCanvas canvas, GlyphRun run, SKTypeface typeface) using var font = new SKFont(typeface, run.Size) { ScaleX = run.ScaleX, + SkewX = run.SkewX, + Embolden = run.Embolden, Subpixel = true, Edging = SKFontEdging.Antialias, Hinting = SKFontHinting.None, diff --git a/src/Filee.Engines/Rendering/PdfPageWriter.cs b/src/Filee.Engines/Rendering/PdfPageWriter.cs index 1d110e4b4..2eff6afaa 100644 --- a/src/Filee.Engines/Rendering/PdfPageWriter.cs +++ b/src/Filee.Engines/Rendering/PdfPageWriter.cs @@ -89,24 +89,26 @@ private static void WritePages(Stream output, IReadOnlyList pages, /// One subset typeface per font the pages use. private static SubsetTypefaces SubsetFonts(IReadOnlyList pages, CancellationToken cancellationToken) { - var glyphs = new Dictionary>(ReferenceEqualityComparer.Instance); + var glyphs = new Dictionary Glyphs, HashSet CodePoints)>(ReferenceEqualityComparer.Instance); foreach (var page in pages) { foreach (var item in page.Items) { if (item is not GlyphItem { Run: var run }) continue; - if (!glyphs.TryGetValue(run.Font, out var set)) - glyphs[run.Font] = set = []; + if (!glyphs.TryGetValue(run.Font, out var used)) + glyphs[run.Font] = used = ([], []); foreach (var glyph in run.Glyphs) - set.Add(glyph); + used.Glyphs.Add(glyph); + foreach (var rune in run.Text.EnumerateRunes()) + used.CodePoints.Add(rune.Value); } } var result = new SubsetTypefaces(); foreach (var (font, used) in glyphs) { cancellationToken.ThrowIfCancellationRequested(); - result.Add(font, SubsetTypeface(font, used, cancellationToken)); + result.Add(font, SubsetTypeface(font, used.Glyphs, used.CodePoints, cancellationToken)); } return result; } @@ -115,11 +117,11 @@ private static SubsetTypefaces SubsetFonts(IReadOnlyList pages, Can /// A typeface over the subset of ; the whole font when it can't be subset (Skia then /// embeds it whole or as a Type 3 font, still with its text). /// - private static SKTypeface SubsetTypeface(PageFont font, IEnumerable glyphs, CancellationToken cancellationToken = default) + private static SKTypeface SubsetTypeface(PageFont font, IEnumerable glyphs, IReadOnlySet codePoints, CancellationToken cancellationToken = default) { if (!FontSubsetter.CanSubset(font.Sfnt, out _)) return font.Typeface; - var subset = FontSubsetter.Subset(font.Sfnt, glyphs, cancellationToken).Write(); + var subset = FontSubsetter.Subset(font.Sfnt, glyphs, cancellationToken, codePoints).Write(); return SKTypeface.FromData(SKData.CreateCopy(subset)) ?? font.Typeface; } diff --git a/src/Filee.Engines/Typesetting/FileeRenderer.cs b/src/Filee.Engines/Typesetting/FileeRenderer.cs new file mode 100644 index 000000000..b0e44d3ad --- /dev/null +++ b/src/Filee.Engines/Typesetting/FileeRenderer.cs @@ -0,0 +1,123 @@ +// Filee's own document renderer, as far as the layout engine goes today: each section's pages with its paper size and +// margins, paragraphs laid out by ParagraphLayouter and stacked down the body, and a new page where the next line +// doesn't fit or a paragraph asks for one; headings become the PDF outline. +// +// This flow stands in for the page composer (Project 4: "Layout engine: page composer"), which brings headers and +// footers, columns, keep-with-next and widow rules, notes and objects. Until then a table's cells are laid out one +// after another as paragraphs across the text width, so their text is there, and objects are left out. + +using Filee.Engines.Documents; +using Filee.Engines.Hwp.Hwpx; +using Filee.Engines.Rendering; + +namespace Filee.Engines.Typesetting; + +internal static class FileeRenderer +{ + /// A4 with 한글's default margins, for sections without a page of their own. + private static readonly HPage DefaultPage = new(59528, 84186, 8504, 8504, 5669, 4252, 4252, 4252, 0); + + /// The document as a PDF. + public static byte[] ToPdf(HDocument document, CancellationToken cancellationToken = default) + { + using var shaper = new Shaper(); + var outline = new List(); + var pages = Pages(document, new FontResolver(), shaper, outline, cancellationToken); + using var output = new MemoryStream(); + PdfPageWriter.Write(output, pages, new PdfDocumentInfo { Title = document.Title, Outline = outline }, cancellationToken: cancellationToken); + return output.ToArray(); + } + + /// The document's pages; headings are added to . + public static List Pages(HDocument document, FontResolver fonts, Shaper shaper, List outline, + CancellationToken cancellationToken = default) + { + var guard = new LayoutGuard(DocumentLimits.Default, cancellationToken); + var layouter = new ParagraphLayouter(fonts, shaper, document.Kinsoku); + var numbers = new ListNumbers(); + var pages = new List(); + PageContent? page = null; + var empty = true; + float y = 0; + + foreach (var section in document.Sections) + { + var size = section.Page ?? DefaultPage; + var left = (size.Left + size.Gutter) / 100f; + var top = (size.Top + size.Header) / 100f; + var bottom = (size.Height - size.Bottom - size.Footer) / 100f; + var width = size.TextWidth / 100f; + + void NewPage() + { + if (page is not null) + guard.PageFinished(empty); + page = new PageContent(size.Width / 100f, size.Height / 100f); + pages.Add(page); + y = top; + empty = true; + } + + void Paragraph(HParagraph paragraph) + { + guard.Paragraph(); + if (paragraph.PageBreakBefore && !empty) + NewPage(); + var marker = paragraph.List is { Numbered: true } list ? numbers.Next(list) : null; + var box = layouter.Layout(paragraph, width, marker); + y += box.SpaceBefore; + for (var i = 0; i < box.Lines.Count; i++) + { + var line = box.Lines[i]; + if (!empty && y + line.Height > bottom + 0.01f) + NewPage(); + if (i == 0 && paragraph.HeadingLevel > 0) + outline.Add(new OutlineEntry(Title(paragraph), pages.Count - 1, y, paragraph.HeadingLevel - 1)); + line.Draw(page!, left, y); + y += line.Height; + empty = false; + } + y += box.SpaceAfter; + } + + void Blocks(IEnumerable blocks) + { + foreach (var block in blocks) + { + switch (block) + { + case HParagraph paragraph: + Paragraph(paragraph); + break; + case HTable table: + if (table.PageBreakBefore && !empty) + NewPage(); + foreach (var cell in table.Rows.SelectMany(r => r.Cells)) + Blocks(cell.Blocks); + Blocks(table.Caption); + break; + } + } + } + + NewPage(); + Blocks(section.Blocks); + } + if (page is null) + { + page = new PageContent(DefaultPage.Width / 100f, DefaultPage.Height / 100f); + pages.Add(page); + } + guard.PageFinished(empty); + return pages; + } + + private static string Title(HParagraph paragraph) => + string.Concat(paragraph.Inlines.Select(i => i switch + { + HText text => text.Text, + HLink link => string.Concat(link.Content.OfType().Select(t => t.Text)), + HTab => " ", + _ => "", + })).Trim(); +} diff --git a/src/Filee.Engines/Typesetting/LineBox.cs b/src/Filee.Engines/Typesetting/LineBox.cs new file mode 100644 index 000000000..b4b56daa0 --- /dev/null +++ b/src/Filee.Engines/Typesetting/LineBox.cs @@ -0,0 +1,150 @@ +// The output of paragraph layout: a paragraph's lines, each with its glyph runs, decorations, links and bookmarks +// placed relative to the line (x from the left edge of the column, y from the baseline, downwards). The page +// composer stacks lines and draws each at its place on a page. + +using Filee.Engines.Rendering; +using SkiaSharp; + +namespace Filee.Engines.Typesetting; + +/// A laid out paragraph. +/// Space above the first line, in points. +/// Space below the last line, in points. +internal sealed record ParagraphBox(float SpaceBefore, float SpaceAfter, IReadOnlyList Lines) +{ + /// The paragraph's whole height with its spacing. + public float Height => SpaceBefore + Lines.Sum(l => l.Height) + SpaceAfter; +} + +/// +/// Glyphs placed on a line: are relative to the line's origin (column edge, baseline); +/// are offsets into as in . +/// +internal sealed record PlacedRun(PageFont Font, float Size, ushort[] Glyphs, SKPoint[] Positions, string Text, int[] Clusters) +{ + public SKColor Color { get; init; } = SKColors.Black; + + public float ScaleX { get; init; } = 1; + + public bool Embolden { get; init; } + + public bool Slant { get; init; } + + public GlyphRun At(float x, float y) => new(Font, Size, Glyphs, [.. Positions.Select(p => new SKPoint(p.X + x, p.Y + y))], Text, Clusters) + { + Color = Color, + ScaleX = ScaleX, + Embolden = Embolden, + SkewX = Slant ? GlyphRun.SyntheticSlant : 0, + }; +} + +internal enum DecorationKind +{ + /// A filled rectangle behind the text (shade, highlight). + Fill, + + /// A line (underline, strike-through). + Line, + + /// A dotted line (tab leader). + Dotted, + + /// A dashed line (tab leader). + Dashed, +} + +/// +/// A rectangle or line on a line, relative to the line's origin: Top and Bottom bound a fill, a line runs +/// at Top with thickness Bottom - Top. +/// +internal readonly record struct Decoration(DecorationKind Kind, float Left, float Right, float Top, float Bottom, SKColor Color); + +/// A link area on a line: from Left to Right, the line's height. +internal readonly record struct LineLink(float Left, float Right, string Target); + +/// One line of a paragraph. +internal sealed class LineBox +{ + /// First character of the line in the paragraph's text. + public required int Start { get; init; } + + /// End of the line's characters (its trailing spaces and line break included). + public required int End { get; init; } + + /// Distance from this line's top to the next line's top, in points (the line pitch). + public required float Height { get; init; } + + /// Distance from the line's top to its baseline. + public required float Baseline { get; init; } + + /// Height of the text itself (ascent and descent). + public required float TextHeight { get; init; } + + /// Left edge of the line's content from the column edge. + public required float Left { get; init; } + + /// Width of the line's content without its trailing spaces. + public required float Width { get; init; } + + public IReadOnlyList Runs { get; init; } = []; + + public IReadOnlyList Decorations { get; init; } = []; + + public IReadOnlyList Links { get; init; } = []; + + public IReadOnlyList<(float X, string Name)> Bookmarks { get; init; } = []; + + /// Draws the line with its top-left corner (column edge, line top) at (, ). + public void Draw(PageContent page, float x, float top) + { + var baseline = top + Baseline; + var fills = Decorations.Where(d => d.Kind == DecorationKind.Fill).ToList(); + if (fills.Count > 0) + page.Draw(canvas => DrawDecorations(canvas, fills, x, baseline)); + foreach (var run in Runs) + page.DrawGlyphs(run.At(x, baseline)); + var lines = Decorations.Where(d => d.Kind != DecorationKind.Fill).ToList(); + if (lines.Count > 0) + page.Draw(canvas => DrawDecorations(canvas, lines, x, baseline)); + foreach (var link in Links) + { + var area = new SKRect(x + link.Left, top, x + link.Right, top + Height); + if (link.Target.StartsWith('#')) + page.LinkToDestination(area, link.Target[1..]); + else + page.LinkToUrl(area, link.Target); + } + foreach (var (position, name) in Bookmarks) + page.Destination(new SKPoint(x + position, top), name); + } + + private static void DrawDecorations(SKCanvas canvas, List decorations, float x, float y) + { + using var paint = new SKPaint { IsAntialias = true }; + foreach (var d in decorations) + { + paint.Color = d.Color; + if (d.Kind == DecorationKind.Fill) + { + paint.Style = SKPaintStyle.Fill; + canvas.DrawRect(new SKRect(x + d.Left, y + d.Top, x + d.Right, y + d.Bottom), paint); + continue; + } + var thickness = Math.Max(0.25f, d.Bottom - d.Top); + float[]? dashes = d.Kind switch + { + DecorationKind.Dotted => [thickness, thickness * 2], + DecorationKind.Dashed => [thickness * 4, thickness * 2], + _ => null, + }; + using var effect = dashes is null ? null : SKPathEffect.CreateDash(dashes, 0); + paint.Style = SKPaintStyle.Stroke; + paint.StrokeWidth = thickness; + paint.PathEffect = effect; + var lineY = y + d.Top + thickness / 2; + canvas.DrawLine(x + d.Left, lineY, x + d.Right, lineY, paint); + paint.PathEffect = null; + } + } +} diff --git a/src/Filee.Engines/Typesetting/LineBreaker.cs b/src/Filee.Engines/Typesetting/LineBreaker.cs new file mode 100644 index 000000000..bccf265ee --- /dev/null +++ b/src/Filee.Engines/Typesetting/LineBreaker.cs @@ -0,0 +1,345 @@ +// Line break opportunities by the Unicode line breaking algorithm (UAX #14, Unicode 16.0, rules LB1–LB31), with the +// tailorings documents ask for: +// - 한글 "어절": Hangul syllables and jamo stay together like letters of a word, so a Korean line breaks only where a +// Latin one would (spaces, after hyphens ...). "글자" is UAX #14's own: a break between any two syllables. +// - Breaking Latin words anywhere (한글 breakLatinWord BREAK_WORD, Word without w:wordWrap): letters and digits of +// Latin, Greek and Cyrillic text break like ideographs. Hyphenation needs dictionaries Filee doesn't have, so it +// keeps words whole. +// - Characters a document forbids at the start or end of a line (Word w:noLineBreaksBefore / After) on top of the +// East Asian rules UAX #14 already has (closing punctuation, small kana and iteration marks don't start a line). +// +// SA characters (Thai, Lao, Khmer, Myanmar) are resolved by LB1's default: marks join their base, letters are AL, so a +// Thai phrase breaks only at its spaces until there is a dictionary. + +using System.Globalization; + +namespace Filee.Engines.Typesetting; + +/// What may happen between two characters. +internal enum LineBreak : byte +{ + /// The line must not break here. + None, + + /// The line may break here. + Allowed, + + /// The line must break here (after a paragraph or line separator, a line feed ...). + Mandatory, +} + +/// Tailorings of the line breaking rules. +/// Hangul breaks only between words (한글 어절), not between syllables (글자). +/// Latin, Greek and Cyrillic words may break between any two letters. +/// Characters that may not start a line, besides UAX #14's. +/// Characters that may not end a line, besides UAX #14's. +internal sealed record LineBreakRules( + bool KeepHangulWords = false, + bool BreakLatinWords = false, + string? NotAtLineStart = null, + string? NotAtLineEnd = null) +{ + public static readonly LineBreakRules Default = new(); +} + +/// Finds where lines may and must break (UAX #14). +internal static class LineBreaker +{ + private const int DottedCircle = 0x25CC; + + /// + /// What may happen before each UTF-16 index of : result[i] is the break between + /// text[i - 1] and text[i], result[text.Length] the end of the text (always mandatory) and + /// result[0] the start (never). Indices inside a surrogate pair or a grapheme's marks are . + /// + public static LineBreak[] Breaks(string text, LineBreakRules? rules = null) + { + rules ??= LineBreakRules.Default; + var result = new LineBreak[text.Length + 1]; + if (text.Length == 0) + return result; + result[text.Length] = LineBreak.Mandatory; + + var units = Units(text, rules); + for (var j = 1; j < units.Count; j++) + result[units[j].Start] = Between(units, j); + + if (rules.NotAtLineStart is { Length: > 0 } || rules.NotAtLineEnd is { Length: > 0 }) + { + foreach (var unit in units) + { + if (unit.Start > 0 && result[unit.Start] == LineBreak.Allowed && Contains(rules.NotAtLineStart, text, unit.Start)) + result[unit.Start] = LineBreak.None; + } + for (var j = 0; j + 1 < units.Count; j++) + { + var next = units[j + 1].Start; + if (result[next] == LineBreak.Allowed && Contains(rules.NotAtLineEnd, text, next - units[j].Length, units[j].Length)) + result[next] = LineBreak.None; + } + } + return result; + } + + /// Whether the code point at (or the unit ending there) is in . + private static bool Contains(string? set, string text, int index, int length = 0) + { + if (string.IsNullOrEmpty(set)) + return false; + var codePoint = UnicodeData.CodePointAt(text, length > 0 ? LastCodePointStart(text, index, length) : index); + return set.EnumerateRunes().Any(r => r.Value == codePoint); + } + + private static int LastCodePointStart(string text, int start, int length) => + length >= 2 && char.IsSurrogatePair(text[start + length - 2], text[start + length - 1]) ? start + length - 2 : start + length - 1; + + // ───────────────────────── Units ───────────────────────── + + /// + /// A character with the combining marks LB9 attaches to it: the rules see its class only. Length covers the + /// marks, so no break falls between them. + /// + private record struct Unit(int Start, int Length, LineBreakClass Class, bool EastAsian, bool InitialQuote, bool FinalQuote, + bool PictographicUnassigned, bool Dotted, bool Hyphen2010, bool EndsWithZwj); + + private static List Units(string text, LineBreakRules rules) + { + var units = new List(text.Length); + for (var i = 0; i < text.Length;) + { + var codePoint = UnicodeData.CodePointAt(text, i); + var size = codePoint > 0xFFFF ? 2 : 1; + var cls = Resolve(codePoint, rules); + // LB9: X (CM | ZWJ)* is X, for any X but BK, CR, LF, NL, SP and ZW. LB10: other marks are AL. + if (cls is LineBreakClass.CM or LineBreakClass.ZWJ && units.Count > 0 && + units[^1].Class is not (LineBreakClass.BK or LineBreakClass.CR or LineBreakClass.LF or LineBreakClass.NL or LineBreakClass.SP or LineBreakClass.ZW)) + { + var last = units[^1]; + units[^1] = last with { Length = last.Length + size, EndsWithZwj = cls == LineBreakClass.ZWJ }; + } + else + { + var isZwj = cls == LineBreakClass.ZWJ; + if (cls is LineBreakClass.CM or LineBreakClass.ZWJ) + cls = LineBreakClass.AL; + var category = CharUnicodeInfo.GetUnicodeCategory(codePoint); + units.Add(new Unit(i, size, cls, + UnicodeData.GetEastAsianWidth(codePoint) is EastAsianWidth.Fullwidth or EastAsianWidth.Wide or EastAsianWidth.Halfwidth, + category == UnicodeCategory.InitialQuotePunctuation, + category == UnicodeCategory.FinalQuotePunctuation, + category == UnicodeCategory.OtherNotAssigned && IsPictographicBlock(codePoint), + codePoint == DottedCircle, + codePoint == 0x2010, + isZwj)); + } + i += size; + } + return units; + } + + /// LB1 and the tailorings: the class the rules work with. + private static LineBreakClass Resolve(int codePoint, LineBreakRules rules) + { + var cls = UnicodeData.GetLineBreak(codePoint); + cls = cls switch + { + LineBreakClass.AI or LineBreakClass.SG or LineBreakClass.XX => LineBreakClass.AL, + LineBreakClass.SA => CharUnicodeInfo.GetUnicodeCategory(codePoint) is UnicodeCategory.NonSpacingMark or UnicodeCategory.SpacingCombiningMark + ? LineBreakClass.CM + : LineBreakClass.AL, + LineBreakClass.CJ => LineBreakClass.NS, + _ => cls, + }; + if (rules.KeepHangulWords && cls is LineBreakClass.H2 or LineBreakClass.H3 or LineBreakClass.JL or LineBreakClass.JV or LineBreakClass.JT) + return LineBreakClass.AL; + if (rules.BreakLatinWords && cls is LineBreakClass.AL or LineBreakClass.NU && + UnicodeData.GetScript(codePoint) is UnicodeScript.Latin or UnicodeScript.Greek or UnicodeScript.Cyrillic or UnicodeScript.Common && + (cls == LineBreakClass.NU || char.IsLetter(char.ConvertFromUtf32(codePoint), 0))) + return LineBreakClass.ID; + return cls; + } + + /// Blocks whose unassigned code points are Extended_Pictographic (reserved for emoji). + private static bool IsPictographicBlock(int codePoint) => codePoint is >= 0x1F000 and <= 0x1FAFF or >= 0x1FC00 and <= 0x1FFFD; + + // ───────────────────────── Rules ───────────────────────── + + /// The break between units[j - 1] and units[j], by rules LB4–LB31. + private static LineBreak Between(List units, int j) + { + var a = units[j - 1]; + var b = units[j]; + var A = a.Class; + var B = b.Class; + var next = j + 1 < units.Count ? units[j + 1].Class : (LineBreakClass?)null; + + // LB4, LB5: hard line breaks. + if (A == LineBreakClass.BK) + return LineBreak.Mandatory; + if (A == LineBreakClass.CR && B == LineBreakClass.LF) + return LineBreak.None; + if (A is LineBreakClass.CR or LineBreakClass.LF or LineBreakClass.NL) + return LineBreak.Mandatory; + // LB6, LB7. + if (B is LineBreakClass.BK or LineBreakClass.CR or LineBreakClass.LF or LineBreakClass.NL or LineBreakClass.SP or LineBreakClass.ZW) + return LineBreak.None; + + // The last unit before the spaces that end at a (LB8, LB14–LB17 look through them). + var k = j - 1; + while (k > 0 && units[k].Class == LineBreakClass.SP) + k--; + var beforeSpaces = units[k]; + + // LB8: ZW SP* ÷. LB8a: ZWJ ×. + if (beforeSpaces.Class == LineBreakClass.ZW) + return LineBreak.Allowed; + if (a.EndsWithZwj) + return LineBreak.None; + // LB11: × WJ, WJ ×. LB12: GL ×. LB12a: [^SP BA HY] × GL. + if (A == LineBreakClass.WJ || B == LineBreakClass.WJ || A == LineBreakClass.GL) + return LineBreak.None; + if (B == LineBreakClass.GL && A is not (LineBreakClass.SP or LineBreakClass.BA or LineBreakClass.HY)) + return LineBreak.None; + // LB13: × CL, × CP, × EX, × SY. + if (B is LineBreakClass.CL or LineBreakClass.CP or LineBreakClass.EX or LineBreakClass.SY) + return LineBreak.None; + // LB14: OP SP* ×. + if (beforeSpaces.Class == LineBreakClass.OP) + return LineBreak.None; + // LB15a: (sot | BK | CR | LF | NL | OP | QU | GL | SP | ZW) [Pi & QU] SP* ×. + if (beforeSpaces is { Class: LineBreakClass.QU, InitialQuote: true } && + (k == 0 || units[k - 1].Class is LineBreakClass.BK or LineBreakClass.CR or LineBreakClass.LF or LineBreakClass.NL or LineBreakClass.OP + or LineBreakClass.QU or LineBreakClass.GL or LineBreakClass.SP or LineBreakClass.ZW)) + return LineBreak.None; + // LB15b: × [Pf & QU] (SP | GL | WJ | CL | QU | CP | EX | IS | SY | BK | CR | LF | NL | ZW | eot). + if (b is { Class: LineBreakClass.QU, FinalQuote: true } && + next is null or LineBreakClass.SP or LineBreakClass.GL or LineBreakClass.WJ or LineBreakClass.CL or LineBreakClass.QU or LineBreakClass.CP + or LineBreakClass.EX or LineBreakClass.IS or LineBreakClass.SY or LineBreakClass.BK or LineBreakClass.CR or LineBreakClass.LF + or LineBreakClass.NL or LineBreakClass.ZW) + return LineBreak.None; + // LB15c: SP ÷ IS NU. LB15d: × IS. + if (A == LineBreakClass.SP && B == LineBreakClass.IS && next == LineBreakClass.NU) + return LineBreak.Allowed; + if (B == LineBreakClass.IS) + return LineBreak.None; + // LB16: (CL | CP) SP* × NS. LB17: B2 SP* × B2. + if (beforeSpaces.Class is LineBreakClass.CL or LineBreakClass.CP && B == LineBreakClass.NS) + return LineBreak.None; + if (beforeSpaces.Class == LineBreakClass.B2 && B == LineBreakClass.B2) + return LineBreak.None; + // LB18: SP ÷. + if (A == LineBreakClass.SP) + return LineBreak.Allowed; + // LB19: × [QU - Pi], [QU - Pf] ×. + if (B == LineBreakClass.QU && !b.InitialQuote || A == LineBreakClass.QU && !a.FinalQuote) + return LineBreak.None; + // LB19a: quotation marks bind unless East Asian text is on both sides. + if (B == LineBreakClass.QU && (!a.EastAsian || j + 1 >= units.Count || !units[j + 1].EastAsian)) + return LineBreak.None; + if (A == LineBreakClass.QU && (!b.EastAsian || j < 2 || !units[j - 2].EastAsian)) + return LineBreak.None; + // LB20: ÷ CB, CB ÷. + if (A == LineBreakClass.CB || B == LineBreakClass.CB) + return LineBreak.Allowed; + // LB20a: (sot | BK | CR | LF | NL | SP | ZW | CB | GL) (HY | ‐) × AL. + if ((A == LineBreakClass.HY || a.Hyphen2010) && B == LineBreakClass.AL && + (j < 2 || units[j - 2].Class is LineBreakClass.BK or LineBreakClass.CR or LineBreakClass.LF or LineBreakClass.NL or LineBreakClass.SP + or LineBreakClass.ZW or LineBreakClass.CB or LineBreakClass.GL)) + return LineBreak.None; + // LB21: × BA, × HY, × NS, BB ×. + if (B is LineBreakClass.BA or LineBreakClass.HY or LineBreakClass.NS || A == LineBreakClass.BB) + return LineBreak.None; + // LB21a: HL (HY | [BA - EastAsian]) × [^HL]. + if (j >= 2 && units[j - 2].Class == LineBreakClass.HL && (A == LineBreakClass.HY || A == LineBreakClass.BA && !a.EastAsian) && B != LineBreakClass.HL) + return LineBreak.None; + // LB21b: SY × HL. LB22: × IN. + if (A == LineBreakClass.SY && B == LineBreakClass.HL || B == LineBreakClass.IN) + return LineBreak.None; + // LB23: (AL | HL) × NU, NU × (AL | HL). + if (IsLetter(A) && B == LineBreakClass.NU || A == LineBreakClass.NU && IsLetter(B)) + return LineBreak.None; + // LB23a: PR × (ID | EB | EM), (ID | EB | EM) × PO. + if (A == LineBreakClass.PR && B is LineBreakClass.ID or LineBreakClass.EB or LineBreakClass.EM || + A is LineBreakClass.ID or LineBreakClass.EB or LineBreakClass.EM && B == LineBreakClass.PO) + return LineBreak.None; + // LB24: (PR | PO) × (AL | HL), (AL | HL) × (PR | PO). + if (A is LineBreakClass.PR or LineBreakClass.PO && IsLetter(B) || IsLetter(A) && B is LineBreakClass.PR or LineBreakClass.PO) + return LineBreak.None; + if (Numeric(units, j, next)) + return LineBreak.None; + // LB26: Korean syllable blocks. + if (A == LineBreakClass.JL && B is LineBreakClass.JL or LineBreakClass.JV or LineBreakClass.H2 or LineBreakClass.H3 || + A is LineBreakClass.JV or LineBreakClass.H2 && B is LineBreakClass.JV or LineBreakClass.JT || + A is LineBreakClass.JT or LineBreakClass.H3 && B == LineBreakClass.JT) + return LineBreak.None; + // LB27: Korean syllable blocks with prefixes and postfixes. + if (IsKorean(A) && B == LineBreakClass.PO || A == LineBreakClass.PR && IsKorean(B)) + return LineBreak.None; + // LB28: (AL | HL) × (AL | HL). + if (IsLetter(A) && IsLetter(B)) + return LineBreak.None; + // LB28a: Brahmic orthographic syllables (Aksara). + if (A == LineBreakClass.AP && IsAksara(b) || + IsAksara(a) && B is LineBreakClass.VF or LineBreakClass.VI || + j >= 2 && IsAksara(units[j - 2]) && A == LineBreakClass.VI && (B == LineBreakClass.AK || b.Dotted) || + IsAksara(a) && IsAksara(b) && next == LineBreakClass.VF) + return LineBreak.None; + // LB29: IS × (AL | HL). + if (A == LineBreakClass.IS && IsLetter(B)) + return LineBreak.None; + // LB30: (AL | HL | NU) × [OP - EastAsian], [CP - EastAsian] × (AL | HL | NU). + if ((IsLetter(A) || A == LineBreakClass.NU) && B == LineBreakClass.OP && !b.EastAsian || + A == LineBreakClass.CP && !a.EastAsian && (IsLetter(B) || B == LineBreakClass.NU)) + return LineBreak.None; + // LB30a: regional indicators pair up. + if (A == LineBreakClass.RI && B == LineBreakClass.RI) + { + var count = 0; + for (var i = j - 1; i >= 0 && units[i].Class == LineBreakClass.RI; i--) + count++; + if (count % 2 == 1) + return LineBreak.None; + } + // LB30b: EB × EM, [Extended_Pictographic & Cn] × EM. + if (B == LineBreakClass.EM && (A == LineBreakClass.EB || a.PictographicUnassigned)) + return LineBreak.None; + // LB31. + return LineBreak.Allowed; + } + + /// LB25: no break inside numbers such as "$(12.5)", "-3", "1,000%". + private static bool Numeric(List units, int j, LineBreakClass? next) + { + var A = units[j - 1].Class; + var B = units[j].Class; + if (B is LineBreakClass.PO or LineBreakClass.PR or LineBreakClass.NU) + { + // NU (SY | IS)* (CL | CP)? × (PO | PR) and NU (SY | IS)* × NU. + var i = j - 1; + if (B != LineBreakClass.NU && units[i].Class is LineBreakClass.CL or LineBreakClass.CP) + i--; + while (i >= 0 && units[i].Class is LineBreakClass.SY or LineBreakClass.IS) + i--; + if (i >= 0 && units[i].Class == LineBreakClass.NU) + return true; + } + if (A is LineBreakClass.PO or LineBreakClass.PR) + { + // (PO | PR) × NU, (PO | PR) × OP NU, (PO | PR) × OP IS NU. + if (B == LineBreakClass.NU) + return true; + if (B == LineBreakClass.OP && (next == LineBreakClass.NU || + next == LineBreakClass.IS && j + 2 < units.Count && units[j + 2].Class == LineBreakClass.NU)) + return true; + } + // HY × NU, IS × NU. + return A is LineBreakClass.HY or LineBreakClass.IS && B == LineBreakClass.NU; + } + + private static bool IsLetter(LineBreakClass cls) => cls is LineBreakClass.AL or LineBreakClass.HL; + + private static bool IsKorean(LineBreakClass cls) => + cls is LineBreakClass.JL or LineBreakClass.JV or LineBreakClass.JT or LineBreakClass.H2 or LineBreakClass.H3; + + private static bool IsAksara(Unit unit) => unit.Class is LineBreakClass.AK or LineBreakClass.AS || unit.Dotted; +} diff --git a/src/Filee.Engines/Typesetting/ParagraphLayout.cs b/src/Filee.Engines/Typesetting/ParagraphLayout.cs new file mode 100644 index 000000000..1ae53d194 --- /dev/null +++ b/src/Filee.Engines/Typesetting/ParagraphLayout.cs @@ -0,0 +1,784 @@ +// Lays out a paragraph into lines (LineBox) for a column of a given width. +// +// 1. Itemise: the text (ParagraphText) splits into items that each shape in one go: one style (the character slot's +// font, size, spacing ...), one font (FontResolver's fallback for characters the font lacks), one script, one +// direction. Tabs and line breaks are items of their own. +// 2. Measure: each item is shaped (Shaper) and every grapheme cluster gets its advance, letter spacing included. +// 3. Break: lines fill greedily up to the column width, breaking at the last opportunity LineBreaker allows (한글 and +// Word break the same way); spaces and tabs at a line end hang past it. A word wider than the line breaks between +// clusters. 한글's 최소 공백 (condense) lets a line keep a word that fits once its spaces shrink. With lineWrap +// KEEP or SQUEEZE the paragraph stays on one line, squeezed to fit with SQUEEZE. +// 4. Place: each line's part of every item is shaped again with its context, so a break inside an Arabic word or +// after an Indic cluster is shaped as it ends; tabs get their width from the stops (left, right, centre, decimal, +// with leaders); alignment adds the remaining width before the line (right, centre) or between its spaces +// (justify; between clusters where a line has no spaces) or its clusters (distribute, 한글 배분, the last line too). +// 5. Line height: by the rule the paragraph carries. 한글's rules (percent, fixed, at least, between lines) measure +// from the font size, the text taking 85% of it above the baseline as 한글 lays out (its stored line segments put +// the baseline at 0.85 × the text height); Word's multiple, and 한글 "글꼴에 어울리는 줄 높이", measure from the +// font's own ascent and descent (OS/2 win metrics, as Word on Windows). +// +// Bidirectional text gets a simplified UBA: strong characters and numbers set the levels, neutrals between two runs of +// one direction take it and the paragraph's otherwise; parts are reordered by level (rule L2). The full algorithm +// (explicit embeddings, isolates, brackets) is a later milestone, as is dictionary breaking for Thai and its kin. +// Tab positions are measured from the paragraph's left indent, as 한글 measures them; a hanging indent is a tab stop +// on the first line (the text after a list number starts there). +// +// 본 제품은 한글과컴퓨터의 한/글 문서 파일(.hwp) 공개 문서를 참고하여 개발하였습니다. + +using Filee.Engines.Hwp.Hwpx; +using SkiaSharp; + +namespace Filee.Engines.Typesetting; + +/// Lays out paragraphs; one per layout job, sharing its font resolver and shaper (not thread-safe). +internal sealed class ParagraphLayouter +{ + /// Distance between the default tab stops, in points (한글's 40 pt). + private const float DefaultTabInterval = 40; + + /// Part of the text height above the baseline in 한글's line layout. + private const float HwpAscent = 0.85f; + + private const float Epsilon = 0.01f; + + private readonly FontResolver _fonts; + private readonly Shaper _shaper; + private readonly string? _notAtLineStart; + private readonly string? _notAtLineEnd; + private readonly Dictionary<(string? Family, FontStyle Style, string Language), ResolvedFont> _resolved = []; + + /// The document's own characters forbidden at a line's start or end (Word settings). + public ParagraphLayouter(FontResolver fonts, Shaper shaper, IReadOnlyList? kinsoku = null) + { + _fonts = fonts; + _shaper = shaper; + if (kinsoku is { Count: > 0 }) + { + _notAtLineStart = string.Concat(kinsoku.Select(k => k.NotAtLineStart)); + _notAtLineEnd = string.Concat(kinsoku.Select(k => k.NotAtLineEnd)); + } + } + + /// Lays out in a column points wide. + /// The list item's number or bullet, or null. + public ParagraphBox Layout(HParagraph paragraph, float width, string? marker = null) + { + var format = paragraph.Format; + var source = ParagraphText.From(paragraph, marker); + var text = source.Text; + var language = FontSubstitution.Language(text); + var rightToLeft = format.RightToLeft ?? FirstStrongIsRightToLeft(text); + var items = Itemize(source, language, Levels(text, rightToLeft)); + var (advances, starts) = Measure(text, items); + + var breaking = format.Layout?.Breaking; + var rules = new LineBreakRules( + KeepHangulWords: breaking?.NonLatin == HNonLatinBreak.KeepWord, + BreakLatinWords: breaking?.Latin == HLatinBreak.BreakWord, + _notAtLineStart, + _notAtLineEnd); + var breaks = LineBreaker.Breaks(text, rules); + var paragraphLayout = new Settings( + format.Align ?? HAlign.Left, + rightToLeft, + (format.Left ?? 0) / 100f, + (format.Right ?? 0) / 100f, + (format.FirstLine ?? 0) / 100f, + format.Tabs ?? [], + breaking?.LineWrap ?? HLineWrap.Break, + Math.Clamp(breaking?.Condense ?? 0, 0, 100) / 100f, + format.Layout?.LineHeight ?? HLineHeight.From(format.LineSpacing ?? new HLineSpacing(HLineSpacingKind.Percent, 160)), + format.Layout?.FontLineHeight == true); + var mark = MarkStyle(paragraph, language); + + var lines = new List(); + var start = 0; + do + { + var first = lines.Count == 0; + var available = width - paragraphLayout.Left - paragraphLayout.Right - (first ? paragraphLayout.FirstLine : 0); + var (end, forced) = paragraphLayout.Wrap == HLineWrap.Break + ? Fit(text, breaks, advances, starts, start, available, x => NextStop(paragraphLayout, first, x), paragraphLayout.Condense) + : (text.Length, true); + lines.Add(Place(source, items, start, end, forced, first, width, paragraphLayout, mark)); + start = end; + } + while (start < text.Length); + // A line break at the end of a paragraph leaves an empty last line, as in 한글 and Word. + if (text.Length > 0 && text[^1] == ParagraphText.LineSeparator) + lines.Add(Place(source, items, text.Length, text.Length, true, false, width, paragraphLayout, mark)); + + return new ParagraphBox((format.Before ?? 0) / 100f, (format.After ?? 0) / 100f, lines); + } + + /// The paragraph settings the line steps share (lengths in points). + private sealed record Settings(HAlign Align, bool RightToLeft, float Left, float Right, float FirstLine, IReadOnlyList Tabs, + HLineWrap Wrap, float Condense, HLineHeight LineHeight, bool FontLineHeight); + + // ───────────────────────── Items ───────────────────────── + + private enum ItemKind + { + Text, + Tab, + Break, + } + + /// Text shaped in one go, or a tab or line break. + /// Bidirectional embedding level: odd is right to left. + private sealed record Item(int Start, int Length, ItemKind Kind, TextStyle Style, ResolvedFont Font, UnicodeScript Script, int Level, string? Link) + { + public int End => Start + Length; + + public bool RightToLeft => (Level & 1) == 1; + } + + private List Itemize(ParagraphText source, string language, int[] levels) + { + var text = source.Text; + var items = new List(); + foreach (var span in source.Spans) + { + foreach (var (start, length, style) in source.Styles(span, language)) + { + var fontLanguage = style.Language.Length > 0 ? style.Language : language; + var primary = Font(style.Family, style.Style, fontLanguage); + foreach (var run in _fonts.Fallback(text, start, length, primary, style.Style, fontLanguage)) + Split(text, run, style, levels, span.Link, items); + } + } + return items; + } + + /// Splits a font run into items of one kind, level and script. + private static void Split(string text, FontRun run, TextStyle style, int[] levels, string? link, List items) + { + var scripts = new UnicodeScript[run.Length]; + var current = UnicodeScript.Common; + for (var i = 0; i < run.Length; i++) + { + var script = UnicodeData.GetScript(UnicodeData.CodePointAt(text, run.Start + i)); + if (char.IsLowSurrogate(text[run.Start + i]) || script is UnicodeScript.Common or UnicodeScript.Inherited) + script = current; + scripts[i] = current = script; + } + // Common characters before the run's first script take it. + var firstScript = scripts.FirstOrDefault(s => s != UnicodeScript.Common); + for (var i = 0; i < run.Length && scripts[i] == UnicodeScript.Common; i++) + scripts[i] = firstScript; + + var itemStart = run.Start; + for (var i = run.Start; i < run.End; i++) + { + var kind = KindOf(text[i]); + var boundary = i + 1 == run.End || kind != ItemKind.Text || KindOf(text[i + 1]) != ItemKind.Text || + levels[i + 1] != levels[i] || scripts[i + 1 - run.Start] != scripts[i - run.Start] && !char.IsLowSurrogate(text[i + 1]); + if (!boundary) + continue; + items.Add(new Item(itemStart, i + 1 - itemStart, kind, style, run.Font, scripts[itemStart - run.Start], levels[itemStart], link)); + itemStart = i + 1; + } + } + + private static ItemKind KindOf(char c) => c switch + { + '\t' => ItemKind.Tab, + ParagraphText.LineSeparator => ItemKind.Break, + _ => ItemKind.Text, + }; + + private ResolvedFont Font(string? family, FontStyle style, string language) + { + var key = (family, style, language); + if (!_resolved.TryGetValue(key, out var font)) + { + _resolved[key] = font = _fonts.Resolve(family, style, language) + ?? _fonts.Resolve(null, style, language) + ?? throw new InvalidOperationException("No installed font can draw the document's text."); + } + return font; + } + + /// The style of the paragraph mark, which gives an empty line its height. + private static TextStyle MarkStyle(HParagraph paragraph, string language) + { + var format = paragraph.Inlines.Count == 0 ? paragraph.MarkFormat : paragraph.Inlines.OfType().FirstOrDefault()?.Format ?? paragraph.MarkFormat; + return format.Layout?.WordFonts is { } word + ? ParagraphText.WordStyle(format, word, WordFontSlot.Ascii, language) + : ParagraphText.HwpStyle(format, HLanguage.Hangul, language); + } + + private static readonly FontFeature[] NoKerning = [new("kern", 0)]; + + private ShapedRun Shape(string text, Item item, int start, int length) => + _shaper.Shape(item.Font.Font, text, start, length, new ShapeOptions( + item.RightToLeft ? TextDirection.RightToLeft : TextDirection.LeftToRight, + item.Script == UnicodeScript.Common ? null : item.Script, + item.Style.Language.Length > 0 ? item.Style.Language : null, + item.Style.Kerning ? null : NoKerning)); + + /// The space added after a character whose advance is . + private static float Spacing(TextStyle style, float advance) => style.LetterSpacing + style.SpacingRatio * advance; + + /// A glyph's advance in points before the style's scale: a space's as the style sets it. + private static float Advance(ShapedRun run, int glyph, string text, int start, TextStyle style) => + style.SpaceWidth is { } space && text[start + run.Clusters[glyph]] == ' ' + ? space + : run.XAdvances[glyph] * style.Size / run.UnitsPerEm; + + /// Per UTF-16 index: the advance of the cluster starting there (0 elsewhere) and whether one starts there. + private (float[] Advances, bool[] Starts) Measure(string text, List items) + { + var advances = new float[text.Length]; + var starts = new bool[text.Length]; + foreach (var item in items) + { + if (item.Kind != ItemKind.Text) + { + starts[item.Start] = true; + continue; + } + var run = Shape(text, item, item.Start, item.Length); + for (var g = 0; g < run.Glyphs.Length; g++) + { + var index = item.Start + run.Clusters[g]; + starts[index] = true; + advances[index] += Advance(run, g, text, item.Start, item.Style) * item.Style.Scale; + } + for (var i = item.Start; i < item.End; i++) + { + if (starts[i]) + advances[i] += Spacing(item.Style, advances[i]); + } + } + return (advances, starts); + } + + // ───────────────────────── Breaking ───────────────────────── + + /// Where the line starting at ends, and whether it had to (a line break, the end). + private static (int End, bool Forced) Fit(string text, LineBreak[] breaks, float[] advances, bool[] starts, int start, float available, + Func nextStop, float condense) + { + var pen = new Pen(); + var lastBreak = -1; + float spaces = 0; + for (var i = start; i < text.Length; i++) + { + if (i > start && breaks[i] == LineBreak.Mandatory) + return (i, true); + if (!starts[i]) + continue; + if (i > start && breaks[i] == LineBreak.Allowed) + lastBreak = i; + var c = text[i]; + if (c == '\t') + { + pen.Tab(nextStop(pen.X)); + continue; + } + var width = advances[i]; + if (IsSpace(c)) + { + pen.Add(width, c); + if (IsStretchable(c)) + spaces += width; + continue; + } + var after = pen.After(width); + if (i > start && after > available + Epsilon && !(condense > 0 && after - spaces * condense <= available + Epsilon)) + return (lastBreak > start ? lastBreak : i, false); + pen.Add(width, c); + } + return (text.Length, true); + } + + /// Spaces that hang at a line end and take no part in its width. + private static bool IsSpace(char c) => c is ' ' or (char)0x3000 or ParagraphText.LineSeparator; + + /// Spaces justification widens and 최소 공백 narrows. + private static bool IsStretchable(char c) => c is ' ' or (char)0x3000; + + private readonly record struct TabStop(float X, HTabKind Kind, string Leader); + + /// The first tab stop after (line coordinates: 0 is where the line starts). + private static TabStop NextStop(Settings settings, bool firstLine, float x) + { + // Stops are measured from the left indent; the first line starts FirstLine from it. + var indent = firstLine ? settings.FirstLine : 0; + TabStop? best = null; + foreach (var tab in settings.Tabs) + { + var position = tab.Position / 100f - indent; + if (position > x + Epsilon && (best is null || position < best.Value.X)) + best = new TabStop(position, tab.Kind, tab.Leader); + } + if (firstLine && settings.FirstLine < 0 && -settings.FirstLine > x + Epsilon && (best is null || -settings.FirstLine < best.Value.X)) + best = new TabStop(-settings.FirstLine, HTabKind.Left, "NONE"); + if (best is { } found) + return found; + var next = (MathF.Floor((x + indent) / DefaultTabInterval + Epsilon) + 1) * DefaultTabInterval - indent; + return new TabStop(next, HTabKind.Left, "NONE"); + } + + /// + /// A line's width as it fills: text after a right, centre or decimal tab moves the tab's end so the text ends at, + /// centres on or puts its decimal point at the stop, as long as the text after it doesn't reach back past the tab. + /// + private sealed class Pen + { + private float _fixed; + private TabStop? _tab; + private float _segment; + private float _beforeDecimal = -1; + + /// Where the line's content ends. + public float X => Gap() + _fixed + _segment; + + public float After(float width) + { + _segment += width; + var x = X; + _segment -= width; + return x; + } + + public void Add(float width, char c) + { + if (_tab is { Kind: HTabKind.Decimal } && _beforeDecimal < 0 && c == '.') + _beforeDecimal = _segment; + _segment += width; + } + + /// Starts the text after a tab; returns the width of the previous pending tab, now known. + public float Tab(TabStop stop) + { + var gap = Gap(); + var x = X; + _fixed = stop.Kind == HTabKind.Left ? Math.Max(stop.X, x) : x; + _tab = stop.Kind == HTabKind.Left ? null : stop; + _segment = 0; + _beforeDecimal = -1; + return gap; + } + + /// The width of the pending right, centre or decimal tab (0 when there is none). + public float Gap() + { + if (_tab is not { } tab) + return 0; + var start = tab.Kind switch + { + HTabKind.Right => tab.X - _segment, + HTabKind.Center => tab.X - _segment / 2, + _ => tab.X - (_beforeDecimal >= 0 ? _beforeDecimal : _segment), + }; + return Math.Max(0, start - _fixed); + } + } + + // ───────────────────────── Placing ───────────────────────── + + /// A line's piece of an item. + private sealed class Part(Item item, int start, int length) + { + public Item Item { get; } = item; + public int Start { get; } = start; + public int Length { get; } = length; + public ShapedRun? Run { get; set; } + public float Width { get; set; } + } + + private LineBox Place(ParagraphText source, List items, int start, int end, bool forced, bool firstLine, float width, Settings settings, + TextStyle mark) + { + var text = source.Text; + var contentEnd = end; + while (contentEnd > start && IsSpace(text[contentEnd - 1])) + contentEnd--; + var indent = firstLine ? settings.FirstLine : 0; + var available = width - settings.Left - settings.Right - indent; + + // The line's parts, shaped with their context, and each cluster's advance. + var parts = new List(); + var advances = new float[contentEnd - start]; + var spacing = new float[contentEnd - start]; + var starts = new bool[contentEnd - start]; + foreach (var item in items) + { + var from = Math.Max(item.Start, start); + var to = Math.Min(item.End, contentEnd); + if (from >= to) + continue; + var part = new Part(item, from, to - from); + parts.Add(part); + if (item.Kind != ItemKind.Text) + { + starts[from - start] = true; + continue; + } + part.Run = Shape(text, item, from, to - from); + for (var g = 0; g < part.Run.Glyphs.Length; g++) + { + var index = from - start + part.Run.Clusters[g]; + starts[index] = true; + advances[index] += Advance(part.Run, g, text, from, item.Style) * item.Style.Scale; + } + for (var i = from - start; i < to - start; i++) + { + if (!starts[i]) + continue; + spacing[i] = Spacing(item.Style, advances[i]); + advances[i] += spacing[i]; + } + } + + // Tabs in logical order, then the natural width. + var pen = new Pen(); + var tabs = new Dictionary(); + var pendingTab = -1; + for (var i = start; i < contentEnd; i++) + { + if (!starts[i - start]) + continue; + if (text[i] == '\t') + { + var before = pen.X; + var stop = NextStop(settings, firstLine, before); + var gap = pen.Tab(stop); + if (pendingTab >= 0) + tabs[pendingTab] = tabs[pendingTab] with { Width = gap }; + // A left tab's width is known now, the others' once the text after them is. + tabs[i] = (stop, stop.Kind == HTabKind.Left ? pen.X - before : 0); + pendingTab = stop.Kind == HTabKind.Left ? -1 : i; + } + else + { + pen.Add(advances[i - start], text[i]); + } + } + if (pendingTab >= 0) + tabs[pendingTab] = tabs[pendingTab] with { Width = pen.Gap() }; + var natural = pen.X; + + // Alignment: the space left over goes before the line or into it. + var extra = available - natural; + var squeeze = 1f; + var gaps = new float[contentEnd - start]; + var align = settings.Align; + if (align == HAlign.Justify && forced) + align = HAlign.Left; + var clusterIndices = Enumerable.Range(start, contentEnd - start).Where(i => starts[i - start]).ToList(); + var spaces = clusterIndices.Where(i => IsStretchable(text[i])).ToList(); + if (settings.Wrap == HLineWrap.Squeeze && extra < 0 && natural > 0) + { + squeeze = available / natural; + extra = 0; + } + else if (extra < 0 && settings.Condense > 0 && spaces.Count > 0) + { + // Spaces shrink by up to the condense share of their width. + var shrink = Math.Max(extra, -spaces.Sum(i => advances[i - start]) * settings.Condense); + foreach (var i in spaces) + gaps[i - start] = shrink / spaces.Count; + extra -= shrink; + } + else if (extra > 0 && align == HAlign.Justify) + { + var stretch = spaces.Count > 0 ? spaces : clusterIndices.Take(clusterIndices.Count - 1).ToList(); + foreach (var i in stretch) + gaps[i - start] = extra / stretch.Count; + if (stretch.Count > 0) + extra = 0; + } + else if (extra > 0 && align == HAlign.Distribute && clusterIndices.Count > 1) + { + foreach (var i in clusterIndices.Take(clusterIndices.Count - 1)) + gaps[i - start] = extra / (clusterIndices.Count - 1); + extra = 0; + } + var offset = Math.Max(0, (align, settings.RightToLeft) switch + { + (HAlign.Center, _) => extra / 2, + (HAlign.Right, false) or (HAlign.Left or HAlign.Justify or HAlign.Distribute, true) => extra, + _ => 0f, + }); + // A right-to-left paragraph's start indent is on the right. + var lineLeft = settings.RightToLeft ? settings.Right : settings.Left + indent; + + // Glyph positions, parts in visual order; the runs are kept in logical order, so the PDF's text reads in the + // order it was written whatever the direction. + var runs = new List<(int Start, PlacedRun Run)>(); + var fills = new List(); + var strokes = new List(); + var links = new List(); + var x = lineLeft + offset; + foreach (var part in Visual(parts)) + { + var x0 = x; + if (part.Item.Kind == ItemKind.Tab) + { + var (stop, tabWidth) = tabs.GetValueOrDefault(part.Start); + tabWidth *= squeeze; + if (stop.Leader is not (null or "" or "NONE") && tabWidth > 0) + strokes.Add(Leader(stop, x, x + tabWidth, part.Item.Style)); + x += tabWidth; + } + else if (part.Run is { } run) + { + var style = part.Item.Style; + var scale = style.Size / run.UnitsPerEm; + var scaleX = style.Scale * squeeze; + var positions = new SKPoint[run.Glyphs.Length]; + for (var g = 0; g < run.Glyphs.Length; g++) + { + if (g > 0 && run.Clusters[g] != run.Clusters[g - 1]) + x += (spacing[part.Start - start + run.Clusters[g - 1]] + gaps[part.Start - start + run.Clusters[g - 1]]) * squeeze; + positions[g] = new SKPoint(x + run.XOffsets[g] * scale * scaleX, -style.Rise - run.YOffsets[g] * scale); + x += Advance(run, g, text, part.Start, style) * scaleX; + } + if (run.Glyphs.Length > 0) + x += (spacing[part.Start - start + run.Clusters[^1]] + gaps[part.Start - start + run.Clusters[^1]]) * squeeze; + runs.Add((part.Start, new PlacedRun(part.Item.Font.Font, style.Size, run.Glyphs, positions, text.Substring(part.Start, part.Length), run.Clusters) + { + Color = style.Color, + ScaleX = scaleX, + Embolden = part.Item.Font.SyntheticBold, + Slant = part.Item.Font.SyntheticItalic, + })); + Decorate(part, x0, x, fills, strokes); + } + if (part.Item.Link is { } target) + { + if (links.Count > 0 && links[^1].Target == target && Math.Abs(links[^1].Right - x0) < Epsilon) + links[^1] = links[^1] with { Right = x }; + else + links.Add(new LineLink(x0, x, target)); + } + } + + var (height, baseline, textHeight) = LineHeight(parts, settings, mark); + var bookmarks = source.Bookmarks.Where(b => b.Position >= start && (b.Position < end || b.Position == end && end == text.Length)) + .Select(b => (lineLeft + offset, b.Name)).ToList(); + return new LineBox + { + Start = start, + End = end, + Height = height, + Baseline = baseline, + TextHeight = textHeight, + Left = lineLeft + offset, + Width = x - lineLeft - offset, + Runs = [.. runs.OrderBy(r => r.Start).Select(r => r.Run)], + Decorations = [.. fills, .. strokes], + Links = links, + Bookmarks = bookmarks, + }; + } + + private static Decoration Leader(TabStop stop, float from, float to, TextStyle style) + { + var thickness = Math.Max(0.5f, style.FontSize / 14); + var kind = stop.Leader switch + { + "DOT" or "CIRCLE" => DecorationKind.Dotted, + "DASH" or "LONG_DASH" or "DASH_DOT" or "DASH_DOT_DOT" => DecorationKind.Dashed, + _ => DecorationKind.Line, + }; + // A little room on either side, as 한글 and Word leave between the leader and the text; at the middle of the + // lowercase letters. + var pad = Math.Min(style.FontSize / 4, (to - from) / 4); + var middle = -style.FontSize / 4; + return new Decoration(kind, from + pad, to - pad, middle - thickness / 2, middle + thickness / 2, style.Color); + } + + /// Shade, underline and strike-through of a part drawn from to . + private void Decorate(Part part, float x0, float x1, List fills, List strokes) + { + var style = part.Item.Style; + if (style.Shade is null && !style.Underline && !style.Strike) + return; + var metrics = _shaper.Metrics(part.Item.Font.Font); + var scale = style.Size / metrics.UnitsPerEm; + var thickness = metrics.UnderlineThickness > 0 ? metrics.UnderlineThickness * scale : style.Size / 14; + if (style.Shade is { } shade) + fills.Add(new Decoration(DecorationKind.Fill, x0, x1, -metrics.Ascender * scale - style.Rise, -metrics.Descender * scale - style.Rise, shade)); + if (style.Underline) + { + var position = metrics.UnderlinePosition != 0 ? -metrics.UnderlinePosition * scale : style.Size / 8; + strokes.Add(new Decoration(DecorationKind.Line, x0, x1, position - style.Rise, position - style.Rise + thickness, style.Color)); + } + if (style.Strike) + { + var strike = metrics.StrikeoutThickness > 0 ? metrics.StrikeoutThickness * scale : thickness; + var position = metrics.StrikeoutPosition != 0 ? metrics.StrikeoutPosition * scale : style.Size * 0.3f; + strokes.Add(new Decoration(DecorationKind.Line, x0, x1, -position - style.Rise, -position - style.Rise + strike, style.Color)); + } + } + + /// The line's pitch, the distance from its top to the baseline and the height of its text. + private (float Height, float Baseline, float TextHeight) LineHeight(List parts, Settings settings, TextStyle mark) + { + var rule = settings.LineHeight; + var fromFont = settings.FontLineHeight || rule.Rule == HLineHeightRule.Multiple; + float ascent = 0, descent = 0; + var any = false; + foreach (var part in parts) + { + if (part.Item.Kind != ItemKind.Text) + continue; + // 한글's line height follows the font size alone; raised or lowered text takes room only with font metrics. + var (a, d) = Extent(part.Item.Style, part.Item.Font, fromFont); + var rise = fromFont ? part.Item.Style.Rise : 0; + ascent = Math.Max(ascent, a + rise); + descent = Math.Max(descent, d - rise); + any = true; + } + if (!any) + { + var font = Font(mark.Family, mark.Style, mark.Language); + (ascent, descent) = Extent(mark, font, fromFont); + } + var textHeight = ascent + descent; + var value = rule.Value / 100f; + var height = rule.Rule switch + { + HLineHeightRule.Percent => textHeight * rule.Value / 100f, + HLineHeightRule.Multiple => textHeight * rule.Value / 240f, + HLineHeightRule.Exact => value, + HLineHeightRule.AtLeast => Math.Max(value, textHeight), + _ => textHeight + value, + }; + return (Math.Max(0, height), ascent, textHeight); + } + + /// Ascent and descent of a style: 한글's share of the size, or the font's own metrics. + private (float Ascent, float Descent) Extent(TextStyle style, ResolvedFont font, bool fromFont) + { + if (!fromFont) + return (style.FontSize * HwpAscent, style.FontSize * (1 - HwpAscent)); + var metrics = _shaper.Metrics(font.Font); + var scale = style.Size / metrics.UnitsPerEm; + if (metrics.WinAscent > 0 || metrics.WinDescent > 0) + return (metrics.WinAscent * scale, metrics.WinDescent * scale); + return (metrics.Ascender * scale, (metrics.LineGap - metrics.Descender) * scale); + } + + // ───────────────────────── Direction ───────────────────────── + + /// True when the first strong character is right to left (UAX #9 rules P2, P3). + private static bool FirstStrongIsRightToLeft(string text) + { + for (var i = 0; i < text.Length; i++) + { + switch (UnicodeData.GetBidiClass(UnicodeData.CodePointAt(text, i))) + { + case BidiClass.L: + return false; + case BidiClass.R or BidiClass.AL: + return true; + } + } + return false; + } + + /// + /// The embedding level of each UTF-16 index, simplified: left-to-right letters 0 (2 in a right-to-left paragraph), + /// right-to-left letters 1, numbers 2 after right-to-left text (else as left-to-right letters), neutrals the level of + /// the strong text on both sides when it agrees, else the paragraph's. + /// + private static int[] Levels(string text, bool rightToLeft) + { + var paragraph = rightToLeft ? 1 : 0; + var left = rightToLeft ? 2 : 0; + // Per index: 'L', 'R', 'N' (number after right-to-left text), ' ' (neutral), '+' (same as the previous). + var kinds = new char[text.Length]; + var lastStrong = rightToLeft ? 'R' : 'L'; + for (var i = 0; i < text.Length; i++) + { + if (char.IsLowSurrogate(text[i]) && i > 0) + { + kinds[i] = '+'; + continue; + } + var bidi = UnicodeData.GetBidiClass(UnicodeData.CodePointAt(text, i)); + kinds[i] = bidi switch + { + BidiClass.L => 'L', + BidiClass.R or BidiClass.AL => 'R', + BidiClass.EN => lastStrong == 'R' ? 'N' : 'L', + BidiClass.AN => 'N', + BidiClass.NSM => '+', + _ => ' ', + }; + if (bidi is BidiClass.L or BidiClass.R or BidiClass.AL) + lastStrong = kinds[i]; + } + + var levels = new int[text.Length]; + for (var i = 0; i < text.Length; i++) + { + switch (kinds[i]) + { + case 'L': + levels[i] = left; + break; + case 'R': + levels[i] = 1; + break; + case 'N': + levels[i] = 2; + break; + case '+': + levels[i] = i > 0 ? levels[i - 1] : paragraph; + break; + default: + // Numbers count as right to left for the neutrals around them (UAX #9 N1). + var before = Strong(kinds, i, -1, rightToLeft); + var after = Strong(kinds, i, 1, rightToLeft); + levels[i] = before == after ? before == 'L' ? left : 1 : paragraph; + break; + } + } + return levels; + } + + private static char Strong(char[] kinds, int index, int step, bool rightToLeft) + { + for (var i = index + step; i >= 0 && i < kinds.Length; i += step) + { + if (kinds[i] is 'L') + return 'L'; + if (kinds[i] is 'R' or 'N') + return 'R'; + } + return rightToLeft ? 'R' : 'L'; + } + + /// The parts in visual order (UAX #9 rule L2). + private static List Visual(List parts) + { + if (parts.Count < 2 || parts.All(p => p.Item.Level == 0)) + return parts; + var order = parts.ToList(); + var highest = order.Max(p => p.Item.Level); + var lowestOdd = order.Where(p => p.Item.RightToLeft).Select(p => p.Item.Level).DefaultIfEmpty(1).Min(); + for (var level = highest; level >= lowestOdd; level--) + { + for (var i = 0; i < order.Count;) + { + if (order[i].Item.Level < level) + { + i++; + continue; + } + var j = i; + while (j < order.Count && order[j].Item.Level >= level) + j++; + order.Reverse(i, j - i); + i = j; + } + } + return order; + } +} diff --git a/src/Filee.Engines/Typesetting/ParagraphText.cs b/src/Filee.Engines/Typesetting/ParagraphText.cs new file mode 100644 index 000000000..45451beeb --- /dev/null +++ b/src/Filee.Engines/Typesetting/ParagraphText.cs @@ -0,0 +1,249 @@ +// A paragraph as the layout reads it: the text of its inlines in one string, and for each piece of it the character +// format it is drawn with, then that format resolved per character into what the shaper and the drawing take +// (TextStyle): the font family of the character's slot, size, width, spacing, raise, colour and decorations. +// +// - Tabs are '\t', line breaks U+2028 (a mandatory break for the line breaker), links keep their target on the pieces +// inside them, bookmarks their position. A list item's number or bullet starts the text, followed by a tab when the +// paragraph has a hanging indent (the text then starts at the indent, as in 한글 and Word) and a space otherwise. +// - Objects (pictures, text boxes, shapes), note references and fields belong to later layout milestones and are left +// out; an equation is drawn as its linear text. +// - 한글's language slots: each character takes the font, 장평, 자간, relative size and 글자 위치 of its slot +// (FontSlots.Hwp); a slot without a font takes the format's Latin font (영문, 외국어, 사용자) or East Asian font +// (한글, 한자, 일어, 기호). Word's run fonts: the character's run font (FontSlots.Word), its scale (w:w), absolute +// letter spacing (w:spacing) and raise (w:position). + +using System.Globalization; +using System.Text; +using Filee.Engines.Hwp.Hwpx; +using SkiaSharp; + +namespace Filee.Engines.Typesetting; + +/// A piece of a paragraph's text in one character format. +/// The target of the link the piece is in, or null. +internal readonly record struct TextSpan(int Start, int Length, HCharFormat Format, string? Link) +{ + public int End => Start + Length; +} + +/// How a character is drawn, with every value resolved. +/// Font family the document names; null for the language's default font. +/// Font size in points, relative size and super- / subscript applied. +/// Glyph width (한글 장평, Word w:w): 1 = as designed. +/// Space added after each character, in points (negative = tighter). +/// Baseline shift in points, upwards. +/// The format's own size in points (what 한글's line height is measured from). +/// "ko", "ja", "zh-Hans", "zh-Hant" or "": picks fonts and language forms. +/// Width of a space before , or null for the font's own. +/// The font's kerning applies. +/// +/// Space added after each character as a share of its own advance (한글's 자간), on top of . +/// +internal sealed record TextStyle( + string? Family, + FontStyle Style, + float Size, + float Scale, + float LetterSpacing, + float Rise, + float FontSize, + SKColor Color, + SKColor? Shade, + bool Underline, + bool Strike, + string Language, + float? SpaceWidth = null, + bool Kerning = true, + float SpacingRatio = 0); + +/// A paragraph's text with the character format of each piece. +internal sealed class ParagraphText +{ + /// A line break inside a paragraph (U+2028 LINE SEPARATOR). + public const char LineSeparator = (char)0x2028; + + /// Size of text whose format leaves it open, in points (한글's default). + public const float DefaultSize = 10; + + private ParagraphText(string text, List spans, List<(int Position, string Name)> bookmarks) + { + Text = text; + Spans = spans; + Bookmarks = bookmarks; + } + + public string Text { get; } + + /// Pieces in text order, covering the text without gaps. + public IReadOnlyList Spans { get; } + + /// Bookmarks by the text position they mark. + public IReadOnlyList<(int Position, string Name)> Bookmarks { get; } + + /// The list item's number or bullet, or null. + public static ParagraphText From(HParagraph paragraph, string? marker = null) + { + var text = new StringBuilder(); + var spans = new List(); + var bookmarks = new List<(int, string)>(); + + void Add(string piece, HCharFormat format, string? link) + { + if (piece.Length == 0) + return; + spans.Add(new TextSpan(text.Length, piece.Length, format, link)); + text.Append(piece); + } + + void Inlines(IEnumerable inlines, string? link) + { + foreach (var inline in inlines) + { + switch (inline) + { + case HText run: + // Paragraph and line feeds inside a run are line breaks; other controls are left out. + Add(Clean(run.Text), run.Format, link); + break; + case HTab tab: + Add("\t", tab.Format, link); + break; + case HLineBreak lineBreak: + Add(LineSeparator.ToString(), lineBreak.Format, link); + break; + case HLink nested: + Inlines(nested.Content, nested.Target); + break; + case HBookmark bookmark: + bookmarks.Add((text.Length, bookmark.Name)); + break; + } + } + } + + if (marker is { Length: > 0 }) + { + var format = paragraph.Inlines.OfType().FirstOrDefault()?.Format ?? paragraph.MarkFormat; + Add(marker + ((paragraph.Format.FirstLine ?? 0) < 0 ? "\t" : " "), format, null); + } + Inlines(paragraph.Inlines, null); + return new ParagraphText(text.ToString(), spans, bookmarks); + } + + private static string Clean(string text) + { + if (!text.Any(c => c < ' ')) + return text; + var sb = new StringBuilder(text.Length); + for (var i = 0; i < text.Length; i++) + { + var c = text[i]; + if (c == '\r' && i + 1 < text.Length && text[i + 1] == '\n') + continue; + if (c is '\n' or '\r' or '\v') + sb.Append(LineSeparator); + else if (c == '\t' || c >= ' ') + sb.Append(c); + } + return sb.ToString(); + } + + // ───────────────────────── Styles ───────────────────────── + + /// + /// Splits a piece into runs of one style: by the character's 한글 language slot, or by Word run font when the + /// format has Word's fonts. + /// + /// The paragraph's language ("ko", "ja", "zh-Hans", "zh-Hant", ""), for Han characters. + public List<(int Start, int Length, TextStyle Style)> Styles(TextSpan span, string language) + { + var format = span.Format; + var result = new List<(int, int, TextStyle)>(); + if (format.Layout?.WordFonts is { } word) + { + foreach (var run in FontSlots.WordRuns(Text, span.Start, span.Length, FontSlots.Hint(word.Hint))) + result.Add((run.Start, run.Length, WordStyle(format, word, run.Slot, language))); + } + else + { + foreach (var run in FontSlots.HwpRuns(Text, span.Start, span.Length)) + result.Add((run.Start, run.Length, HwpStyle(format, run.Slot, language))); + } + return result; + } + + /// The style of a character in 한글 language slot . + internal static TextStyle HwpStyle(HCharFormat format, HLanguage slot, string language) + { + var own = format.Layout?.Slots?[slot] ?? default; + var eastAsian = slot is HLanguage.Hangul or HLanguage.Hanja or HLanguage.Japanese or HLanguage.Symbol; + var family = own.Font ?? (eastAsian ? format.EastAsianFont ?? format.Font : format.Font ?? format.EastAsianFont); + var fontSize = (format.Size ?? DefaultSize * 100) / 100f; + var size = fontSize * (own.RelativeSize ?? 100) / 100f; + var scale = (own.Ratio ?? format.Layout?.Scale ?? 100) / 100f; + // 자간 is a share of each character's own width (with 장평 applied), not of the font size: a space or a narrowed + // character closes up less. 한글's stored line layout breaks where this puts the line end. + var spacing = (own.Spacing ?? format.Spacing ?? 0) / 100f; + // 글자 위치: percent of the size, negative raises. + var rise = -(own.Offset ?? 0) / 100f * size + (format.Layout?.Position ?? 0) / 100f; + var slotLanguage = slot switch + { + HLanguage.Hangul => "ko", + HLanguage.Japanese => "ja", + HLanguage.Hanja => language, + _ => "", + }; + return Finish(format, family, fontSize, size, scale, 0, rise, slotLanguage, spacing); + } + + /// The style of a character drawn with Word run font . + internal static TextStyle WordStyle(HCharFormat format, HWordFonts fonts, WordFontSlot slot, string language) + { + var family = fonts.Font(slot) ?? (slot == WordFontSlot.EastAsia ? format.EastAsianFont ?? format.Font : format.Font); + var fontSize = (format.Size ?? DefaultSize * 100) / 100f; + var scale = (format.Layout?.Scale ?? 100) / 100f; + var spacing = format.Layout?.LetterSpacing is { } absolute ? absolute / 100f : (format.Spacing ?? 0) / 100f * fontSize; + var rise = (format.Layout?.Position ?? 0) / 100f; + return Finish(format, family, fontSize, fontSize, scale, spacing, rise, slot == WordFontSlot.EastAsia ? language : ""); + } + + private static TextStyle Finish(HCharFormat format, string? family, float fontSize, float size, float scale, float spacing, float rise, string language, + float spacingRatio = 0) + { + // Super- and subscript: two thirds of the size, raised a third of it or lowered a seventh. + if (format.Superscript == true) + { + rise += size / 3; + size *= 2f / 3; + } + else if (format.Subscript == true) + { + rise -= size / 7; + size *= 2f / 3; + } + size = Math.Max(0.5f, size); + return new TextStyle( + family, + FontStyle.From(format.Bold == true, format.Italic == true), + size, + scale > 0 ? scale : 1, + spacing, + rise, + fontSize, + Color(format.Color) ?? SKColors.Black, + Color(format.Shade), + format.Underline == true, + format.Strike == true, + language, + // 한글 without "글꼴에 어울리는 빈칸": a space is half the size wide. + format.Layout?.FontSpace == false ? size / 2 : null, + format.Layout?.Kerning != false, + spacingRatio); + } + + /// A "#RRGGBB" colour, or null for none or an unreadable one. + internal static SKColor? Color(string? value) => + value is { Length: 7 } && value[0] == '#' && uint.TryParse(value.AsSpan(1), NumberStyles.HexNumber, CultureInfo.InvariantCulture, out var rgb) + ? new SKColor((byte)(rgb >> 16), (byte)(rgb >> 8), (byte)rgb) + : null; +} diff --git a/tests/Filee.Engines.Tests/DocumentModelTests.cs b/tests/Filee.Engines.Tests/DocumentModelTests.cs index e7fd13beb..c1bb24c02 100644 --- a/tests/Filee.Engines.Tests/DocumentModelTests.cs +++ b/tests/Filee.Engines.Tests/DocumentModelTests.cs @@ -196,7 +196,8 @@ public void Hwpx_paragraph_shapes_keep_line_rules_borders_and_line_breaking() var layout = boxed.Layout!; Assert.Equal(new HLineHeight(HLineHeightRule.BetweenLines, 500), layout.LineHeight); Assert.Equal((false, true), (layout.SnapToGrid, layout.FontLineHeight)); - Assert.Equal(new HLineBreaking(HLatinBreak.Hyphenation, HNonLatinBreak.BreakWord, HLineWrap.Squeeze, AutoSpaceLatin: true, AutoSpaceDigits: false, Condense: 25), layout.Breaking); + // breakNonLatinWord BREAK_WORD is 한글's 어절 (whole words): 한글 names the two the other way round. + Assert.Equal(new HLineBreaking(HLatinBreak.Hyphenation, HNonLatinBreak.KeepWord, HLineWrap.Squeeze, AutoSpaceLatin: true, AutoSpaceDigits: false, Condense: 25), layout.Breaking); var border = layout.Border!; Assert.Equal(new HBorder(HBorderStyle.Solid, 0.4, "#0000FF"), border.Borders.Top); Assert.Equal(("#FFF2CC", new HInsets(100, 200, 300, 400), true, true), (border.Fill, border.Offsets, border.Connect, border.IgnoreMargin)); diff --git a/tests/Filee.Engines.Tests/FontSubsetterTests.cs b/tests/Filee.Engines.Tests/FontSubsetterTests.cs index a6c1184cd..101c6f5de 100644 --- a/tests/Filee.Engines.Tests/FontSubsetterTests.cs +++ b/tests/Filee.Engines.Tests/FontSubsetterTests.cs @@ -89,6 +89,24 @@ public void Composite_glyphs_bring_their_components() Assert.Equal(expected.ToSvgPathData(), actual.ToSvgPathData()); } + [Fact] + public void A_glyph_two_characters_share_maps_back_to_the_one_the_text_has() + { + // As CJK fonts map 文 and the Kangxi radical ⽂ to one glyph: a PDF's ToUnicode would take the lower code point. + var source = Load(TrueType); + var map = CmapTable.Read(source.Require("cmap"), out var symbol); + var h = map['H']; + map[0x2F42] = h; + source.Tables["cmap"] = CmapTable.Build(map, symbol); + + var all = CmapTable.Read(FontSubsetter.Subset(source, [h], TestContext.Current.CancellationToken).Require("cmap"), out _); + var text = CmapTable.Read(FontSubsetter.Subset(source, [h], TestContext.Current.CancellationToken, codePoints: new HashSet { 'H' }) + .Require("cmap"), out _); + + Assert.Equal(h, all[0x2F42]); + Assert.Equal(new Dictionary { ['H'] = h }, text); + } + [Fact] public void Subset_keeps_only_what_a_pdf_needs() { diff --git a/tests/Filee.Engines.Tests/LineBreakerTests.cs b/tests/Filee.Engines.Tests/LineBreakerTests.cs new file mode 100644 index 000000000..25d00a332 --- /dev/null +++ b/tests/Filee.Engines.Tests/LineBreakerTests.cs @@ -0,0 +1,113 @@ +// The layout engine's line breaking (Typesetting/LineBreaker.cs): UAX #14 on text written here, and the tailorings +// documents ask for (한글 어절 / 글자, Latin words broken anywhere, characters forbidden at a line's start or end). +// +// The implementation passes all 16,672 cases of Unicode 16.0's LineBreakTest.txt; that file is not kept in the +// repository, so these cases cover each rule family instead. Expected breaks are written into the text: '|' where a +// line may break, '¶' where it must (after the character before it). + +using System.Text; +using Filee.Engines.Typesetting; + +namespace Filee.Engines.Tests; + +public class LineBreakerTests +{ + [Theory] + [InlineData("Hello |world")] + [InlineData("Filee, |the |converter.")] + [InlineData("e-|mail |co-|op")] // LB21: no break before a hyphen; after it, yes + [InlineData("-5 |-mail")] // LB25 HY × NU; LB20a: a hyphen at a word start stays with it + [InlineData("$(12.50) |1,000% |US$5")] // LB25 numbers, LB24 PR × AL + [InlineData("(a) |[b] |{c}")] // LB14 OP ×, LB13 × CP / CL + [InlineData("“Hello,” |she |said")] // LB15a/b: quotes bind to what they quote + [InlineData("a\u00A0b |c\u2060d")] // LB12 GL, LB11 WJ + [InlineData("a\u200B|b")] // LB8 ZW ÷ + [InlineData("e\u0301e\u0301 |x")] // LB9: combining marks stay with their base + [InlineData("👨\u200D👩\u200D👧 |👍🏽|👍🏽")] // LB8a ZWJ ×, LB30b EB × EM + [InlineData("🇰🇷|🇯🇵|🇺🇸")] // LB30a: regional indicators in pairs + [InlineData("ภาษาไทยไม่เว้นวรรค |ระหว่างคำ")] // SA without a dictionary: only at spaces + [InlineData("क्षत्रिय |हिन्दी")] // Devanagari conjuncts: marks join their base + public void Breaks_by_uax_14(string expected) => Assert.Equal(expected, Mark(Unmark(expected))); + + [Theory] + [InlineData("한|글 |문|서|는 |줄|을 |나|눕|니|다.")] // UAX #14: between any two syllables (한글 글자) + [InlineData("\u1100\u1161\u11A8|\u1100\u1161")] // LB26: conjoining jamo form one syllable + [InlineData("文|件|转|换|工|具。|汉|字")] // ideographs break anywhere, never before 。 (LB13) + [InlineData("ちょっ|とー|ず")] // small kana and the prolonged sound mark start no line (CJ resolved to NS) + [InlineData("「漢|字」|と|か")] // LB14 / LB13 corner brackets + public void Breaks_east_asian_text(string expected) => Assert.Equal(expected, Mark(Unmark(expected))); + + [Theory] + [InlineData("a\n¶b")] + [InlineData("a\r\n¶b")] // LB5: CR × LF + [InlineData("a\u2028¶b")] + [InlineData("a \n¶ |b")] + public void Breaks_after_line_separators(string expected) => Assert.Equal(expected, Mark(Unmark(expected))); + + [Fact] + public void The_end_of_the_text_is_a_mandatory_break_and_the_start_none() + { + var breaks = LineBreaker.Breaks("ab"); + Assert.Equal([LineBreak.None, LineBreak.None, LineBreak.Mandatory], breaks); + Assert.Equal([LineBreak.None], LineBreaker.Breaks("")); + } + + [Fact] + public void A_surrogate_pair_is_never_split() + { + var breaks = LineBreaker.Breaks("𠀀𠀁"); + Assert.Equal(LineBreak.None, breaks[1]); + Assert.Equal(LineBreak.Allowed, breaks[2]); + Assert.Equal(LineBreak.None, breaks[3]); + } + + [Theory] + [InlineData("한글 |문서는 |줄을 |나눕니다.")] + [InlineData("Filee는 |한글 |문서를 |PDF로 |바꿉니다.")] + [InlineData("한글|漢|字 |섞어 |쓰기")] // ideographs still break apart from Hangul + public void Keeps_hangul_words_whole(string expected) => + Assert.Equal(expected, Mark(Unmark(expected), new LineBreakRules(KeepHangulWords: true))); + + [Theory] + [InlineData("F|i|l|e|e |l|a|y|s |1|2|3 |п|р|и|в|е|т.")] + [InlineData("(a|b)")] // punctuation keeps its rules + [InlineData("한|글 |w|o|r|d")] + [InlineData("مرحبا |بالعالم")] // Arabic words stay whole: breaking them would break the joining + public void Breaks_latin_words_anywhere(string expected) => + Assert.Equal(expected, Mark(Unmark(expected), new LineBreakRules(BreakLatinWords: true))); + + [Fact] + public void Honours_characters_forbidden_at_the_start_or_end_of_a_line() + { + var rules = new LineBreakRules(NotAtLineStart: "字", NotAtLineEnd: "換"); + Assert.Equal("漢字|変|換文|書", Mark("漢字変換文書", rules)); + } + + [Fact] + public void Forbidden_characters_do_not_cancel_mandatory_breaks() + { + var rules = new LineBreakRules(NotAtLineStart: "b"); + Assert.Equal("a\n¶b", Mark("a\nb", rules)); + } + + /// The text with '|' at each allowed break and '¶' at each mandatory one (but the end). + private static string Mark(string text, LineBreakRules? rules = null) + { + var breaks = LineBreaker.Breaks(text, rules); + var result = new StringBuilder(); + for (var i = 0; i < text.Length; i++) + { + if (i > 0) + { + if (breaks[i] == LineBreak.Allowed) + result.Append('|'); + else if (breaks[i] == LineBreak.Mandatory) + result.Append('¶'); + } + result.Append(text[i]); + } + return result.ToString(); + } + + private static string Unmark(string marked) => marked.Replace("|", "").Replace("¶", ""); +} diff --git a/tests/Filee.Engines.Tests/ParagraphLayoutTests.cs b/tests/Filee.Engines.Tests/ParagraphLayoutTests.cs new file mode 100644 index 000000000..e5c2ae951 --- /dev/null +++ b/tests/Filee.Engines.Tests/ParagraphLayoutTests.cs @@ -0,0 +1,506 @@ +// The layout engine's paragraph layout (Typesetting/ParagraphLayout.cs, ParagraphText.cs): line breaking against the +// column width, 한글 어절 / 글자, alignment, indents, tabs, line height rules, character spacing and the right-to-left +// start. Paragraphs are written here and laid out with the Noto fonts build/fetch-fonts.ps1 puts into +// src/Filee.App/Assets/Fonts (CI fetches them; without them the tests skip), so the results don't depend on the fonts +// a computer has. Expected widths come from the shaper itself, not from numbers kept here. + +using Filee.Engines.Hwp.Hwpx; +using Filee.Engines.Rendering; +using Filee.Engines.Typesetting; +using SkiaSharp; + +namespace Filee.Engines.Tests; + +public sealed class ParagraphLayoutTests : IDisposable +{ + private const string Latin = "Noto Sans"; + private const string Korean = "Noto Sans KR"; + private const string Arabic = "Noto Sans Arabic"; + + private const string English = "Filee lays out every paragraph the way the document asks: line spacing, alignment and the place where each line ends."; + private const string Hangul = "한글 문서는 줄 간격과 정렬에 따라 줄이 나뉘는 자리가 달라집니다. 같은 문단을 몇 번 그려도 줄은 같은 자리에서 끝나야 합니다."; + + private readonly Shaper _shaper = new(); + private readonly ParagraphLayouter _layouter; + private readonly RepositoryFonts _fonts; + + public ParagraphLayoutTests() + { + _fonts = RepositoryFonts.Load(); + _layouter = new ParagraphLayouter(new FontResolver(_fonts), _shaper); + } + + public void Dispose() => _shaper.Dispose(); + + // ───────────────────────── Breaking ───────────────────────── + + [Fact] + public void Lines_cover_the_text_fit_the_column_and_break_after_spaces() + { + var (box, text) = Layout(Paragraph(English), 200); + + Assert.True(box.Lines.Count > 2); + Assert.Equal(text, string.Concat(box.Lines.Select(l => text[l.Start..l.End]))); + foreach (var line in box.Lines) + Assert.True(line.Width <= 200.01f, $"{line.Width} > 200"); + foreach (var line in box.Lines.SkipLast(1)) + Assert.Equal(' ', text[line.End - 1]); + // Greedy: the next word wouldn't have fitted. + var first = box.Lines[0]; + var nextWord = text[first.End..].Split(' ')[0]; + Assert.True(Width(text[first.Start..first.End] + nextWord, Latin) > 200); + } + + [Fact] + public void Korean_breaks_between_syllables_by_default_and_between_words_with_eojeol() + { + var syllables = Layout(Paragraph(Hangul, breaking: new HLineBreaking(NonLatin: HNonLatinBreak.BreakWord)), 180); + var words = Layout(Paragraph(Hangul, breaking: new HLineBreaking(NonLatin: HNonLatinBreak.KeepWord)), 180); + + Assert.All(words.Box.Lines.SkipLast(1), line => Assert.Equal(' ', words.Text[line.End - 1])); + Assert.Contains(syllables.Box.Lines.SkipLast(1), line => syllables.Text[line.End - 1] != ' '); + Assert.True(syllables.Box.Lines.Count <= words.Box.Lines.Count); + } + + [Fact] + public void Latin_words_break_anywhere_when_the_paragraph_says_so() + { + var (box, text) = Layout(Paragraph(English, breaking: new HLineBreaking(Latin: HLatinBreak.BreakWord)), 150); + + Assert.Contains(box.Lines.SkipLast(1), line => text[line.End - 1] != ' '); + Assert.All(box.Lines, line => Assert.True(line.Width <= 150.01f)); + } + + [Fact] + public void A_word_wider_than_the_column_breaks_between_clusters() + { + var (box, text) = Layout(Paragraph("Pneumonoultramicroscopicsilicovolcanoconiosis"), 60); + + Assert.True(box.Lines.Count > 1); + Assert.Equal(text, string.Concat(box.Lines.Select(l => text[l.Start..l.End]))); + Assert.All(box.Lines, line => Assert.True(line.Width <= 60.01f)); + } + + [Fact] + public void A_line_break_ends_the_line_and_a_last_one_leaves_an_empty_line() + { + var paragraph = Paragraph("one"); + paragraph.Inlines.Add(new HLineBreak(Format())); + paragraph.Inlines.Add(new HText("two", Format())); + paragraph.Inlines.Add(new HLineBreak(Format())); + + var (box, text) = Layout(paragraph, 400); + + Assert.Equal(3, box.Lines.Count); + Assert.Equal("two", text[box.Lines[1].Start..box.Lines[1].End].TrimEnd(ParagraphText.LineSeparator)); + Assert.Equal(box.Lines[2].Start, box.Lines[2].End); + Assert.Empty(box.Lines[2].Runs); + } + + [Fact] + public void Condense_keeps_a_word_that_fits_once_the_spaces_shrink() + { + var text = "aaaa aaaa aaaa aaaa aaaa aaaa"; + var natural = Width(text, Latin); + // The last word is 2 pt too wide; five spaces can give that up at 50%. + var width = natural - 2; + + var plain = Layout(Paragraph(text), width); + var condensed = Layout(Paragraph(text, breaking: new HLineBreaking(Condense: 50)), width); + + Assert.Equal(2, plain.Box.Lines.Count); + var line = Assert.Single(condensed.Box.Lines); + Assert.True(line.Width <= width + 0.01f, $"{line.Width} > {width}"); + } + + [Theory] + [InlineData(nameof(HLineWrap.Keep))] + [InlineData(nameof(HLineWrap.Squeeze))] + public void Keep_and_squeeze_hold_the_paragraph_on_one_line(string mode) + { + var wrap = Enum.Parse(mode); + var (box, _) = Layout(Paragraph(English, breaking: new HLineBreaking(LineWrap: wrap)), 200); + + var line = Assert.Single(box.Lines); + if (wrap == HLineWrap.Squeeze) + Assert.InRange(line.Width, 199.9f, 200.01f); + else + Assert.True(line.Width > 200); + } + + // ───────────────────────── Alignment and indents ───────────────────────── + + [Theory] + [InlineData(nameof(HAlign.Left))] + [InlineData(nameof(HAlign.Right))] + [InlineData(nameof(HAlign.Center))] + public void Alignment_places_each_line_in_the_column(string alignment) + { + var align = Enum.Parse(alignment); + var (box, _) = Layout(Paragraph(English, new HParaFormat(Align: align, Left: 1000, Right: 2000)), 300); + + foreach (var line in box.Lines) + { + var expected = align switch + { + HAlign.Left => 10f, + HAlign.Right => 300 - 20 - line.Width, + _ => 10 + (300 - 30 - line.Width) / 2, + }; + Assert.Equal(expected, line.Left, 0.01f); + } + } + + [Fact] + public void Justify_fills_every_line_but_the_last() + { + var (box, _) = Layout(Paragraph(English, new HParaFormat(Align: HAlign.Justify)), 220); + + foreach (var line in box.Lines.SkipLast(1)) + Assert.Equal(220, line.Width, 0.05f); + Assert.True(box.Lines[^1].Width < 219); + Assert.Equal(0, box.Lines[^1].Left, 0.01f); + } + + [Fact] + public void Distribute_fills_the_last_line_too() + { + var (box, _) = Layout(Paragraph(Hangul, new HParaFormat(Align: HAlign.Distribute)), 220); + + Assert.All(box.Lines, line => Assert.Equal(220, line.Width, 0.05f)); + } + + [Fact] + public void A_first_line_indent_moves_the_first_line_and_a_hanging_one_the_others() + { + var indented = Layout(Paragraph(English, new HParaFormat(Left: 1000, FirstLine: 2000)), 250).Box; + var hanging = Layout(Paragraph(English, new HParaFormat(Left: 3000, FirstLine: -2000)), 250).Box; + + Assert.Equal(30, indented.Lines[0].Left, 0.01f); + Assert.Equal(10, indented.Lines[1].Left, 0.01f); + Assert.Equal(10, hanging.Lines[0].Left, 0.01f); + Assert.Equal(30, hanging.Lines[1].Left, 0.01f); + Assert.True(indented.Lines[0].Width <= 250 - 30 + 0.01f); + } + + [Theory] + [InlineData(nameof(HTabKind.Left))] + [InlineData(nameof(HTabKind.Right))] + [InlineData(nameof(HTabKind.Center))] + [InlineData(nameof(HTabKind.Decimal))] + public void A_tab_aligns_the_text_after_it_at_its_stop(string tab) + { + var kind = Enum.Parse(tab); + var paragraph = new HParagraph { Format = new HParaFormat(Tabs: [new HTabStop(15000, kind, "DOT")]) }; + paragraph.Inlines.Add(new HText("Item", Format())); + paragraph.Inlines.Add(new HTab(Format())); + paragraph.Inlines.Add(new HText("123.45", Format())); + + var line = Assert.Single(Layout(paragraph, 400).Box.Lines); + + var (start, end, dot) = Extent(line, "123.45"); + var at = kind switch + { + HTabKind.Left => start, + HTabKind.Right => end, + HTabKind.Center => (start + end) / 2, + _ => dot, + }; + Assert.Equal(150, at, 0.05f); + // The leader runs between "Item" and the number. + var leader = Assert.Single(line.Decorations, d => d.Kind == DecorationKind.Dotted); + Assert.True(leader.Left >= Extent(line, "Item").End && leader.Right <= start); + } + + [Fact] + public void Default_tab_stops_are_every_forty_points_from_the_indent() + { + var paragraph = new HParagraph { Format = new HParaFormat(Left: 1000) }; + paragraph.Inlines.Add(new HText("a", Format())); + paragraph.Inlines.Add(new HTab(Format())); + paragraph.Inlines.Add(new HText("b", Format())); + + var line = Assert.Single(Layout(paragraph, 400).Box.Lines); + + Assert.Equal(10 + 40, Extent(line, "b").Start, 0.05f); + } + + [Fact] + public void A_list_number_hangs_and_the_text_starts_at_the_indent() + { + var paragraph = Paragraph(English, new HParaFormat(Left: 2000, FirstLine: -2000)); + + var (box, text) = Layout(paragraph, 250, marker: "1."); + + Assert.StartsWith("1.\t", text); + Assert.Equal(0, box.Lines[0].Left, 0.01f); + Assert.Equal(20, Extent(box.Lines[0], "Filee").Start, 0.05f); + Assert.Equal(20, box.Lines[1].Left, 0.01f); + } + + // ───────────────────────── Line height ───────────────────────── + + [Theory] + [InlineData(nameof(HLineHeightRule.Percent), 160, 16)] + [InlineData(nameof(HLineHeightRule.Percent), 100, 10)] + [InlineData(nameof(HLineHeightRule.Exact), 2000, 20)] + [InlineData(nameof(HLineHeightRule.AtLeast), 2400, 24)] + [InlineData(nameof(HLineHeightRule.AtLeast), 500, 10)] + [InlineData(nameof(HLineHeightRule.BetweenLines), 500, 15)] + public void Hangul_line_rules_measure_from_the_font_size(string lineRule, int value, float pitch) + { + var rule = Enum.Parse(lineRule); + var (box, _) = Layout(Paragraph(English, new HParaFormat(Layout: new HParaLayout(new HLineHeight(rule, value)))), 200); + + Assert.All(box.Lines, line => Assert.Equal(pitch, line.Height, 0.01f)); + // 한글 puts the baseline at 85% of the text height. + Assert.Equal(8.5f, box.Lines[0].Baseline, 0.01f); + } + + [Theory] + [InlineData(240)] + [InlineData(360)] + public void Word_multiple_line_spacing_measures_from_the_font(int value) + { + var (box, _) = Layout(Paragraph(English, new HParaFormat(Layout: new HParaLayout(new HLineHeight(HLineHeightRule.Multiple, value)))), 200); + + var metrics = _shaper.Metrics(_fonts.Font(Latin)); + var single = (metrics.WinAscent + metrics.WinDescent) * 10f / metrics.UnitsPerEm; + Assert.All(box.Lines, line => Assert.Equal(single * value / 240, line.Height, 0.01f)); + Assert.Equal(metrics.WinAscent * 10f / metrics.UnitsPerEm, box.Lines[0].Baseline, 0.01f); + } + + [Fact] + public void The_largest_text_on_a_line_sets_its_height() + { + var paragraph = Paragraph("small "); + paragraph.Inlines.Add(new HText("BIG", Format(size: 2000))); + + var line = Assert.Single(Layout(paragraph, 400).Box.Lines); + + Assert.Equal(32, line.Height, 0.01f); // 160% of 20 pt + } + + [Fact] + public void An_empty_paragraph_is_one_line_as_high_as_its_mark() + { + var paragraph = new HParagraph { MarkFormat = Format(size: 1500) }; + + var line = Assert.Single(Layout(paragraph, 400).Box.Lines); + + Assert.Equal(24, line.Height, 0.01f); + Assert.Empty(line.Runs); + } + + [Fact] + public void Space_before_and_after_are_kept_apart_from_the_lines() + { + var (box, _) = Layout(Paragraph(English, new HParaFormat(Before: 600, After: 1200)), 200); + + Assert.Equal(6, box.SpaceBefore); + Assert.Equal(12, box.SpaceAfter); + Assert.Equal(6 + 12 + box.Lines.Sum(l => l.Height), box.Height, 0.01f); + } + + // ───────────────────────── Characters ───────────────────────── + + [Fact] + public void Letter_spacing_widens_each_character_by_a_share_of_its_own_width() + { + var plain = Single(Paragraph("abcd")).Width; + var spaced = Single(Paragraph("abcd", format: Format(spacing: 10))).Width; + var narrow = Single(Paragraph("abcd", format: Format(spacing: 10) with { Layout = new HCharLayout(HLanguageSlots.All(new HSlotFormat(Ratio: 50))) })).Width; + + // 한글's 자간: 10% of each character's width, the width ratio applied. + Assert.Equal(plain * 1.1f, spaced, 0.01f); + Assert.Equal(plain / 2 * 1.1f, narrow, 0.01f); + } + + [Fact] + public void Word_letter_spacing_is_an_absolute_distance() + { + var plain = Single(Paragraph("abcd")).Width; + var spaced = Single(Paragraph("abcd", format: Format() with { Layout = new HCharLayout(WordFonts: new HWordFonts(Latin, Latin, Korean, Latin), LetterSpacing: 100) })).Width; + + Assert.Equal(plain + 4, spaced, 0.01f); // 1 pt after each of four characters + } + + [Fact] + public void The_width_ratio_scales_the_glyphs() + { + var plain = Single(Paragraph("abcd")); + var narrow = Single(Paragraph("abcd", format: Format() with { Layout = new HCharLayout(HLanguageSlots.All(new HSlotFormat(Ratio: 50))) })); + + Assert.Equal(plain.Width / 2, narrow.Width, 0.01f); + Assert.Equal(0.5f, narrow.Runs[0].ScaleX, 0.001f); + } + + [Fact] + public void Without_the_fonts_own_space_a_space_is_half_the_size() + { + var format = Format() with { Layout = new HCharLayout(FontSpace: false) }; + + var spaced = Single(Paragraph("a a", format: format)).Width; + var joined = Single(Paragraph("aa", format: format)).Width; + + Assert.Equal(joined + 5, spaced, 0.01f); + } + + [Fact] + public void Superscript_is_smaller_and_raised() + { + var paragraph = Paragraph("x"); + paragraph.Inlines.Add(new HText("2", Format() with { Superscript = true })); + + var line = Single(paragraph); + + var two = line.Runs.Single(r => r.Text == "2"); + Assert.True(two.Size < 10); + Assert.True(two.Positions[0].Y < 0, "raised above the baseline"); + } + + [Fact] + public void Underline_strike_and_shade_are_drawn_under_and_through_the_text() + { + var format = Format() with { Underline = true, Strike = true, Shade = "#FFFF00" }; + + var line = Single(Paragraph("marked", format: format)); + + var fill = Assert.Single(line.Decorations, d => d.Kind == DecorationKind.Fill); + Assert.Equal(new SKColor(255, 255, 0), fill.Color); + var lines = line.Decorations.Where(d => d.Kind == DecorationKind.Line).OrderBy(d => d.Top).ToList(); + Assert.Equal(2, lines.Count); + Assert.True(lines[0].Top < 0, "strike-through above the baseline"); + Assert.True(lines[1].Top > 0, "underline below it"); + Assert.All(line.Decorations, d => Assert.Equal(line.Width, d.Right - d.Left, 0.01f)); + } + + [Fact] + public void Links_cover_their_text() + { + var paragraph = Paragraph("see "); + var link = new HLink("https://filee.sh"); + link.Content.Add(new HText("filee.sh", Format())); + paragraph.Inlines.Add(link); + + var line = Single(paragraph); + + var area = Assert.Single(line.Links); + Assert.Equal("https://filee.sh", area.Target); + var (start, end, _) = Extent(line, "filee.sh"); + Assert.Equal(start, area.Left, 0.01f); + Assert.Equal(end, area.Right, 0.01f); + } + + [Fact] + public void A_right_to_left_paragraph_starts_at_the_right_and_keeps_its_text_in_order() + { + Assert.SkipUnless(_fonts.Has(Arabic), "Noto Sans Arabic is not in src/Filee.App/Assets/Fonts (build/fetch-fonts.ps1)"); + const string text = "يحافظ برنامج التحويل على تنسيق الفقرات ومواضع فواصل الأسطر."; + + var (box, _) = Layout(Paragraph(text, format: Format(font: Arabic)), 150); + + Assert.True(box.Lines.Count > 1); + Assert.All(box.Lines, line => Assert.Equal(150, line.Left + line.Width, 0.05f)); + Assert.Equal(text.Replace(" ", ""), string.Concat(box.Lines.SelectMany(l => l.Runs).Select(r => r.Text)).Replace(" ", "")); + } + + // ───────────────────────── Helpers ───────────────────────── + + private static HCharFormat Format(int size = 1000, int spacing = 0, string font = Latin) => + new(Size: size, Font: font, EastAsianFont: Korean, Spacing: spacing, Color: "#000000"); + + private static HParagraph Paragraph(string text, HParaFormat paragraph = default, HLineBreaking? breaking = null, HCharFormat? format = null) + { + if (breaking is not null) + paragraph = paragraph with { Layout = (paragraph.Layout ?? new HParaLayout()) with { Breaking = breaking } }; + var result = new HParagraph { Format = paragraph, MarkFormat = format ?? Format() }; + result.Inlines.Add(new HText(text, format ?? Format())); + return result; + } + + private (ParagraphBox Box, string Text) Layout(HParagraph paragraph, float width, string? marker = null) + { + Assert.SkipUnless(_fonts.Has(Latin) && _fonts.Has(Korean), "Noto Sans and Noto Sans KR are not in src/Filee.App/Assets/Fonts (build/fetch-fonts.ps1)"); + return (_layouter.Layout(paragraph, width, marker), ParagraphText.From(paragraph, marker).Text); + } + + private LineBox Single(HParagraph paragraph) => Assert.Single(Layout(paragraph, 400).Box.Lines); + + private float Width(string text, string family) => _shaper.Shape(_fonts.Font(family), text).Width(10); + + /// Where the glyphs of start and end on the line, and the x of its '.' (or NaN). + private static (float Start, float End, float Dot) Extent(LineBox line, string text) + { + var run = line.Runs.First(r => r.Text.Contains(text, StringComparison.Ordinal)); + var from = run.Text.IndexOf(text, StringComparison.Ordinal); + using var font = new SKFont(run.Font.Typeface, run.Size) { ScaleX = run.ScaleX, LinearMetrics = true, Subpixel = true }; + float start = float.MaxValue, end = float.MinValue, dot = float.NaN; + for (var g = 0; g < run.Glyphs.Length; g++) + { + var cluster = run.Clusters[g]; + if (cluster < from || cluster >= from + text.Length) + continue; + var x = run.Positions[g].X; + start = Math.Min(start, x); + end = Math.Max(end, x + font.GetGlyphWidths([run.Glyphs[g]])[0]); + if (run.Text[cluster] == '.') + dot = x; + } + return (start, end, dot); + } + + /// The Noto fonts of src/Filee.App/Assets/Fonts as the only fonts there are. + private sealed class RepositoryFonts : IFontCollection + { + private readonly Dictionary> _families = new(StringComparer.OrdinalIgnoreCase); + private readonly Dictionary _fonts = new(StringComparer.OrdinalIgnoreCase); + + public static RepositoryFonts Load() + { + var fonts = new RepositoryFonts(); + var folder = Folder(); + if (folder is null) + return fonts; + (string File, string Family, int Weight)[] files = + [ + ("NotoSans-Regular.ttf", Latin, 400), ("NotoSans-Bold.ttf", Latin, 700), + ("NotoSansKR-Regular.otf", Korean, 400), ("NotoSansKR-Bold.otf", Korean, 700), + ("NotoSansArabic-Regular.ttf", Arabic, 400), + ]; + foreach (var (file, family, weight) in files) + { + var path = Path.Combine(folder, file); + if (!File.Exists(path)) + continue; + var font = PageFont.FromFile(path); + var typeface = font.Typeface; + if (weight == 400) + fonts._fonts[family] = font; + if (!fonts._families.TryGetValue(family, out var faces)) + fonts._families[family] = faces = []; + faces.Add(new FontFace(family, weight, false, c => typeface.ContainsGlyph(c), () => font)); + } + return fonts; + } + + public bool Has(string family) => _fonts.ContainsKey(family); + + public PageFont Font(string family) => _fonts[family]; + + public IReadOnlyList Family(string name) => _families.GetValueOrDefault(name) ?? []; + + public FontFace? ForCharacter(int codePoint, FontStyle style, string? language) => + _families.Values.SelectMany(f => f).FirstOrDefault(f => f.Weight == 400 && f.Has(codePoint)); + + private static string? Folder() + { + for (var dir = new DirectoryInfo(AppContext.BaseDirectory); dir is not null; dir = dir.Parent) + { + if (File.Exists(Path.Combine(dir.FullName, "Filee.slnx"))) + return Path.Combine(dir.FullName, "src", "Filee.App", "Assets", "Fonts"); + } + return null; + } + } +} diff --git a/tests/Filee.Engines.Tests/RenderQuality/DocumentRenderers.cs b/tests/Filee.Engines.Tests/RenderQuality/DocumentRenderers.cs index f037cfcf8..5cff12050 100644 --- a/tests/Filee.Engines.Tests/RenderQuality/DocumentRenderers.cs +++ b/tests/Filee.Engines.Tests/RenderQuality/DocumentRenderers.cs @@ -9,6 +9,7 @@ using Filee.Engines.Hwp; using Filee.Engines.Hwp.Hwpx; using Filee.Engines.Infrastructure; +using Filee.Engines.Typesetting; namespace Filee.Engines.Tests.RenderQuality; @@ -40,14 +41,15 @@ internal sealed record DocumentRenderer(string Name, IReadOnlySet Format internal static class DocumentRenderers { /// - /// Filee's layout engine, measured against ; null until its first milestone registers a - /// renderer here (for example new("filee", Hwpx, (input, ct) => Task.FromResult(FileeRenderer.ToPdf(input.Document, ct)))). + /// Filee's layout engine, measured against : paragraph layout on the interim page flow + /// (Typesetting/FileeRenderer.cs) until the page composer replaces it. Null would skip the compare. /// - public static DocumentRenderer? Candidate => null; + public static DocumentRenderer? Candidate => + new("filee", Hwpx, (input, ct) => Task.FromResult(FileeRenderer.ToPdf(input.Document, ct))); /// Why the differential compare doesn't run while is null. public const string NoCandidate = - "Filee's layout engine has no renderer yet: the first layout milestone registers one in DocumentRenderers.Candidate"; + "Filee's layout engine has no renderer registered in DocumentRenderers.Candidate"; public static readonly IReadOnlySet Hwpx = new HashSet(["hwpx"]); public static readonly IReadOnlySet HwpxAndHwp = new HashSet(["hwpx", "hwp"]); diff --git a/tests/Filee.Engines.Tests/RenderQuality/HangulLineBreakTests.cs b/tests/Filee.Engines.Tests/RenderQuality/HangulLineBreakTests.cs new file mode 100644 index 000000000..6e49ea2e9 --- /dev/null +++ b/tests/Filee.Engines.Tests/RenderQuality/HangulLineBreakTests.cs @@ -0,0 +1,112 @@ +// 한글's own line breaks as the yardstick. An HWPX that 한글 saved stores where each line of a paragraph starts and +// how wide 한글 made the line (hp:linesegarray). For every paragraph of plain text in the HWPX files of +// FILEE_RENDER_CORPUS, Filee's layout engine lays the paragraph out at that width, with the paragraph and character +// shapes of the file, and its line starts are compared with 한글's. +// +// Left out: paragraphs with anything but text (objects, equations, tabs, fields, line breaks), because 한글 counts each +// of those as eight characters in its positions and draws objects at their own widths, and paragraphs whose lines +// differ in width (text wrapped around an object). Fonts that aren't installed are replaced, so their paragraphs can +// break elsewhere. Like the corpus compare, it reports and never fails on the numbers. + +using System.IO.Compression; +using System.Xml.Linq; +using Filee.Engines.Hwp.Hwpx; +using Filee.Engines.Typesetting; + +namespace Filee.Engines.Tests.RenderQuality; + +[Trait("Category", "RenderCorpus")] +public class HangulLineBreakTests +{ + private static readonly XNamespace Hp = "http://www.hancom.co.kr/hwpml/2011/paragraph"; + + [Fact] + public void Line_breaks_agree_with_the_ones_hangul_stored() + { + var corpus = Environment.GetEnvironmentVariable("FILEE_RENDER_CORPUS"); + Assert.SkipWhen(string.IsNullOrEmpty(corpus), "FILEE_RENDER_CORPUS is not set (see build/render-compare.ps1)"); + var files = Directory.EnumerateFiles(corpus, "*.hwpx", SearchOption.AllDirectories).Order().ToList(); + Assert.SkipWhen(files.Count == 0, $"No HWPX in {corpus}"); + var output = TestContext.Current.TestOutputHelper; + + using var shaper = new Shaper(); + var layouter = new ParagraphLayouter(new FontResolver(), shaper); + int paragraphs = 0, exact = 0, lines = 0, agreed = 0, shown = 0; + foreach (var file in files) + { + int fileParagraphs = 0, fileExact = 0; + foreach (var (paragraph, width, stored) in Paragraphs(file)) + { + var box = layouter.Layout(paragraph, width); + var ours = box.Lines.Skip(1).Select(l => l.Start).ToList(); + var same = ours.SequenceEqual(stored); + fileParagraphs++; + fileExact += same ? 1 : 0; + lines += Math.Max(ours.Count, stored.Count) + 1; + agreed += ours.Intersect(stored).Count() + 1; + if (!same && shown++ < 30) + { + var text = string.Concat(paragraph.Inlines.OfType().Select(t => t.Text)); + output?.WriteLine($" {Path.GetFileName(file)}: 한글 [{string.Join(", ", stored)}], Filee [{string.Join(", ", ours)}] at {width:0.##} pt"); + output?.WriteLine($" {text[..Math.Min(120, text.Length)]}"); + } + } + paragraphs += fileParagraphs; + exact += fileExact; + output?.WriteLine($"{Path.GetFileName(file)}: {fileExact} of {fileParagraphs} paragraphs break where 한글 broke them"); + } + output?.WriteLine($"All: {exact} of {paragraphs} paragraphs ({(paragraphs == 0 ? 1 : (double)exact / paragraphs):P1}), " + + $"{agreed} of {lines} line starts ({(lines == 0 ? 1 : (double)agreed / lines):P1})"); + } + + /// + /// The paragraphs of plain text of an HWPX, each with the shapes of the file, the width 한글 laid it out at and the + /// text positions where its second and later lines start. + /// + internal static IEnumerable<(HParagraph Paragraph, float Width, List Starts)> Paragraphs(string path) + { + using var zip = ZipFile.OpenRead(path); + var header = zip.GetEntry("Contents/header.xml"); + if (header is null) + yield break; + XElement head; + using (var stream = header.Open()) + head = XDocument.Load(stream).Root!; + var shapes = HwpxReader.ForHeader(head); + + foreach (var entry in zip.Entries.Where(e => e.FullName.StartsWith("Contents/section", StringComparison.Ordinal)).OrderBy(e => e.FullName)) + { + XDocument section; + using (var stream = entry.Open()) + section = XDocument.Load(stream, LoadOptions.PreserveWhitespace); + foreach (var p in section.Descendants(Hp + "p")) + { + var segments = p.Element(Hp + "linesegarray")?.Elements(Hp + "lineseg").ToList(); + var runs = p.Elements(Hp + "run").ToList(); + if (segments is not { Count: > 0 } || runs.SelectMany(r => r.Elements()).Any(e => e.Name != Hp + "t" || e.HasElements)) + continue; + var paragraph = new HParagraph { Format = shapes.ParagraphShape((string?)p.Attribute("paraPrIDRef") ?? "0") }; + foreach (var run in runs) + { + var format = shapes.CharacterShape((string?)run.Attribute("charPrIDRef") ?? "0"); + foreach (var t in run.Elements(Hp + "t")) + paragraph.Inlines.Add(new HText(t.Value, format)); + } + var length = paragraph.Inlines.Sum(i => ((HText)i).Text.Length); + if (length == 0) + continue; + + // 한글 stores every line from the paragraph's left margin to its right one, indents inside: lines that + // start or end elsewhere go around an object. + int Int(XElement e, string name) => (int?)e.Attribute(name) ?? 0; + var (position, size) = (Int(segments[0], "horzpos"), Int(segments[0], "horzsize")); + if (segments.Any(s => Int(s, "horzpos") != position || Int(s, "horzsize") != size)) + continue; + var starts = segments.Skip(1).Select(s => Int(s, "textpos")).ToList(); + if (starts.Any(s => s <= 0 || s >= length)) + continue; + yield return (paragraph, (position + size + (paragraph.Format.Right ?? 0)) / 100f, starts); + } + } + } +} diff --git a/tests/Filee.Engines.Tests/RenderQuality/render-thresholds.json b/tests/Filee.Engines.Tests/RenderQuality/render-thresholds.json index e24a16bc1..382429eb1 100644 --- a/tests/Filee.Engines.Tests/RenderQuality/render-thresholds.json +++ b/tests/Filee.Engines.Tests/RenderQuality/render-thresholds.json @@ -36,8 +36,28 @@ "script-gu.hwp": { "overlaps": 0.6387 }, "script-th.hwp": { "text": 0.9734, "overlaps": 0.1709 } }, - // Filee's layout engine (DocumentRenderers.Candidate): its own read-back checks and vs-rhwp-* compares. - "filee": {} + // Filee's layout engine (DocumentRenderers.Candidate): its own read-back checks and vs-rhwp-* compares, with + // paragraph layout on the interim page flow (Typesetting/FileeRenderer.cs). + "filee": { + // Korean words: the paragraphs say 어절 (breakNonLatinWord BREAK_WORD, which 한글 names the other way round). + // rhwp still splits some words ("않습니|다.") and its distributed lines elsewhere. + "line-spacing-percent": { "vs-rhwp-lines": 0.875, "vs-rhwp-placed": 0.875, "vs-rhwp-ssim": 0.6268 }, + "line-spacing-fixed": { "vs-rhwp-lines": 0.875, "vs-rhwp-placed": 0.875, "vs-rhwp-ssim": 0.6273 }, + "line-spacing-at-least": { "vs-rhwp-lines": 0.875, "vs-rhwp-placed": 0.875, "vs-rhwp-ssim": 0.6259 }, + "script-ko": { "vs-rhwp-lines": 0.6, "vs-rhwp-placed": 0.6, "vs-rhwp-ssim": 0.4694 }, + // Same lines and baselines; rhwp draws its leaders as short dashes at other heights and puts the text a little + // apart (pictures differ by fractions of a point). + "tabs": { "vs-rhwp-ssim": 0.3844 }, + "lists": { "vs-rhwp-ssim": 0.2697 }, + "script-zh": { "vs-rhwp-ssim": 0.9923 }, + "script-ja": { "vs-rhwp-ssim": 0.9933 }, + // Left to the page composer: tables (cells are stacked as paragraphs for now), headers and footers, notes and + // columns. + "tables": { "vs-rhwp-lines": 0.1538, "vs-rhwp-placed": 0.0769, "vs-rhwp-ssim": 0.1915 }, + "headers-footers": { "vs-rhwp-lines": 0.3333, "vs-rhwp-placed": 0.3333, "vs-rhwp-ssim": 0.5714 }, + "notes": { "vs-rhwp-lines": 0.25, "vs-rhwp-placed": 0.25, "vs-rhwp-ssim": 0.2707 }, + "columns": { "vs-rhwp-lines": 0, "vs-rhwp-placed": 0, "vs-rhwp-ssim": 0.0864 } + } }, "allow": [ // Where rhwp is wrong, Filee's renderer can't agree with it: the compare is reported, never failed.