From e088d09e4705a4348444aaa8ec1b49a07128e063 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 13 Oct 2024 14:18:23 +0100 Subject: [PATCH 01/41] ZIP Export: Started defining format --- dev/docs/portable-zip-file-format.md | 82 ++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 dev/docs/portable-zip-file-format.md diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md new file mode 100644 index 000000000..260735c58 --- /dev/null +++ b/dev/docs/portable-zip-file-format.md @@ -0,0 +1,82 @@ +# Portable ZIP File Format + +BookStack provides exports in a "Portable ZIP" which allows the portable transfer, storage, import & export of BookStack content. +This document details the format used, and is intended for our own internal development use in addition to detailing the format for potential external use-cases (readers, apps, import for other platforms etc...). + +**Note:** This is not a BookStack backup format! This format misses much of the data that would be needed to re-create/restore a BookStack instance. There are existing better alternative options for this use-case. + +## Stability + +Following the goals & ideals of BookStack, stability is very important. We aim for this defined format to be stable and forwards compatible, to prevent breakages in use-case due to changes. Here are the general rules we follow in regard to stability & changes: + +- New features & properties may be added with any release. +- Where reasonably possible, we will attempt to avoid modifications/removals of existing features/properties. +- Where potentially breaking changes do have to be made, these will be noted in BookStack release/update notes. + +The addition of new features/properties alone are not considered as a breaking change to the format. Breaking changes are considered as such where they could impact common/expected use of the existing properties and features we document, they are not considered based upon user assumptions or any possible breakage. For example if your application, using the format, breaks because we added a new property while you hard-coded your application to use the third property (instead of a property name), then that's on you. + +## Format Outline + +The format is intended to be very simple, readable and based on open standards that could be easily read/handled in most common programming languages. +The below outlines the structure of the format: + +- **ZIP archive container** + - **data.json** - Application data. + - **files/** - Directory containing referenced files. + - *...file.ext* + +## References + +TODO - Define how we reference across content: +TODO - References to files from data.json +TODO - References from in-content to file URLs +TODO - References from in-content to in-export content (page cross links within same export). + +## Application Data - `data.json` + +The `data.json` file is a JSON format file which contains all structured data for the export. The properties are as follows: + +- `instance` - [Instance](#instance) Object, optional, details of the export source instance. +- `exported_at` - String, optional, full ISO 8601 datetime of when the export was created. +- `book` - [Book](#book) Object, optional, book export data. +- `chapter` - [Chapter](#chapter) Object, optional, chapter export data. +- `page` - [Page](#page) Object, optional, page export data. + +Either `book`, `chapter` or `page` will exist depending on export type. You'd want to check for each to check what kind of export this is, and if it's an export you can handle. It's possible that other options are added in the future (`books` for a range of books for example) so it'd be wise to specifically check for properties that can be handled, otherwise error to indicate lack of support. + +## Data Objects + +The below details the objects & their properties used in Application Data. + +#### Instance + +These details are mainly informational regarding the exporting BookStack instance from where an export was created from. + +- `version` - String, required, BookStack version of the export source instance. +- `id_ciphertext` - String, required, identifier for the BookStack instance. + +The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This is used as a simple & rough way for a BookStack instance to be able to identify if they were the source (by attempting to decrypt the ciphertext). + +#### Book + +TODO + +#### Chapter + +TODO + +#### Page + +TODO + +#### Image + +TODO + +#### Attachment + +TODO + +#### Tag + +TODO \ No newline at end of file From 1930af91cea9ab830562933f738b1c9d151b0640 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 13 Oct 2024 22:56:22 +0100 Subject: [PATCH 02/41] ZIP Export: Started types in format doc --- dev/docs/portable-zip-file-format.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 260735c58..c4737309f 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -59,11 +59,22 @@ The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This i #### Book -TODO +- `id` - Number, optional, original ID for the book from exported system. +- `name` - String, required, name/title of the book. +- `description_html` - String, optional, HTML description content. +- `chapters` - [Chapter](#chapter) array, optional, chapters within this book. +- `pages` - [Page](#page) array, optional, direct child pages for this book. +- `tags` - [Tag](#tag) array, optional, tags assigned to this book. + +The `pages` are not all pages within the book, just those that are direct children (not in a chapter). To build an ordered mixed chapter/page list for the book, as what you'd see in BookStack, you'd need to combine `chapters` and `pages` together and sort by their `priority` value (low to high). #### Chapter -TODO +- `id` - Number, optional, original ID for the chapter from exported system. +- `name` - String, required, name/title of the chapter. +- `description_html` - String, optional, HTML description content. +- `pages` - [Page](#page) array, optional, pages within this chapter. +- `tags` - [Tag](#tag) array, optional, tags assigned to this chapter. #### Page @@ -79,4 +90,6 @@ TODO #### Tag -TODO \ No newline at end of file +- `name` - String, required, name of the tag. +- `value` - String, optional, value of the tag (can be empty). +- `order` - Number, optional, integer order for the tags (shown low to high). \ No newline at end of file From 42bd07d73325ed468bae2658ba130314f6fe4a21 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 15 Oct 2024 13:57:16 +0100 Subject: [PATCH 03/41] ZIP Export: Continued expanding format doc types --- dev/docs/portable-zip-file-format.md | 57 ++++++++++++++++++++++++---- 1 file changed, 49 insertions(+), 8 deletions(-) diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index c4737309f..dc21bf8e5 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -21,18 +21,38 @@ The format is intended to be very simple, readable and based on open standards t The below outlines the structure of the format: - **ZIP archive container** - - **data.json** - Application data. + - **data.json** - Export data. - **files/** - Directory containing referenced files. - - *...file.ext* + - *file-a* + - *file-b* + - *...* ## References +Some properties in the export data JSON are indicated as `String reference`, and these are direct references to a file name within the `files/` directory of the ZIP. For example, the below book cover is directly referencing a `files/4a5m4a.jpg` within the ZIP which would be expected to exist. + +```json +{ + "book": { + "cover": "4a5m4a.jpg" + } +} +``` + +TODO - Jotting out idea below. +Would need to validate image/attachment paths against image/attachments listed across all pages in export. +Probably good to ensure filenames are ascii-alpha-num. +`[[bsexport:image:an-image-path.png]]` +`[[bsexport:attachment:an-image-path.png]]` +`[[bsexport:page:1]]` +`[[bsexport:chapter:2]]` +`[[bsexport:book:3]]` + TODO - Define how we reference across content: -TODO - References to files from data.json TODO - References from in-content to file URLs TODO - References from in-content to in-export content (page cross links within same export). -## Application Data - `data.json` +## Export Data - `data.json` The `data.json` file is a JSON format file which contains all structured data for the export. The properties are as follows: @@ -62,6 +82,7 @@ The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This i - `id` - Number, optional, original ID for the book from exported system. - `name` - String, required, name/title of the book. - `description_html` - String, optional, HTML description content. +- `cover` - String reference, options, reference to book cover image. - `chapters` - [Chapter](#chapter) array, optional, chapters within this book. - `pages` - [Page](#page) array, optional, direct child pages for this book. - `tags` - [Tag](#tag) array, optional, tags assigned to this book. @@ -73,23 +94,43 @@ The `pages` are not all pages within the book, just those that are direct childr - `id` - Number, optional, original ID for the chapter from exported system. - `name` - String, required, name/title of the chapter. - `description_html` - String, optional, HTML description content. +- `priority` - Number, optional, integer order for when shown within a book (shown low to high). - `pages` - [Page](#page) array, optional, pages within this chapter. - `tags` - [Tag](#tag) array, optional, tags assigned to this chapter. #### Page -TODO +- `id` - Number, optional, original ID for the page from exported system. +- `name` - String, required, name/title of the page. +- `html` - String, optional, page HTML content. +- `markdown` - String, optional, user markdown content for this page. +- `priority` - Number, optional, integer order for when shown within a book (shown low to high). +- `attachments` - [Attachment](#attachment) array, optional, attachments uploaded to this page. +- `images` - [Image](#image) array, optional, images used in this page. +- `tags` - [Tag](#tag) array, optional, tags assigned to this page. + +To define the page content, either `markdown` or `html` should be provided. Ideally these should be limited to the range of markdown and HTML which BookStack supports. + +The page editor type, and edit content will be determined by what content is provided. If non-empty `markdown` is provided, the page will be assumed as a markdown editor page (where permissions allow) and the HTML will be rendered from the markdown content. Otherwise, the provided `html` will be used as editor and display content. #### Image -TODO +- `name` - String, required, name of image. +- `file` - String reference, required, reference to image file. + +File must be an image type accepted by BookStack (png, jpg, gif, webp) #### Attachment -TODO +- `name` - String, required, name of attachment. +- `link` - String, semi-optional, URL of attachment. +- `file` - String reference, semi-optional, reference to attachment file. +- `order` - Number, optional, integer order of the attachments (shown low to high). + +Either `link` or `file` must be present, as that will determine the type of attachment. #### Tag - `name` - String, required, name of the tag. - `value` - String, optional, value of the tag (can be empty). -- `order` - Number, optional, integer order for the tags (shown low to high). \ No newline at end of file +- `order` - Number, optional, integer order of the tags (shown low to high). \ No newline at end of file From 42b9700673e7b2e5a04c9f888a05d98261ed36e3 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 15 Oct 2024 16:14:11 +0100 Subject: [PATCH 04/41] ZIP Exports: Finished up format doc, move files, started builder Moved all existing export related app files into their new own dir. --- app/Exceptions/ZipExportException.php | 7 +++ .../Controllers/BookExportApiController.php | 4 +- .../Controllers/BookExportController.php | 4 +- .../ChapterExportApiController.php | 4 +- .../Controllers/ChapterExportController.php | 4 +- .../Controllers/PageExportApiController.php | 4 +- .../Controllers/PageExportController.php | 4 +- .../Tools => Exports}/ExportFormatter.php | 4 +- .../Tools => Exports}/PdfGenerator.php | 4 +- app/Exports/ZipExportBuilder.php | 48 +++++++++++++++++++ composer.json | 1 + dev/docs/portable-zip-file-format.md | 28 ++++++----- routes/api.php | 26 +++++----- routes/web.php | 27 ++++++----- tests/Entity/ExportTest.php | 2 +- 15 files changed, 118 insertions(+), 53 deletions(-) create mode 100644 app/Exceptions/ZipExportException.php rename app/{Entities => Exports}/Controllers/BookExportApiController.php (95%) rename app/{Entities => Exports}/Controllers/BookExportController.php (95%) rename app/{Entities => Exports}/Controllers/ChapterExportApiController.php (95%) rename app/{Entities => Exports}/Controllers/ChapterExportController.php (96%) rename app/{Entities => Exports}/Controllers/PageExportApiController.php (95%) rename app/{Entities => Exports}/Controllers/PageExportController.php (96%) rename app/{Entities/Tools => Exports}/ExportFormatter.php (98%) rename app/{Entities/Tools => Exports}/PdfGenerator.php (99%) create mode 100644 app/Exports/ZipExportBuilder.php diff --git a/app/Exceptions/ZipExportException.php b/app/Exceptions/ZipExportException.php new file mode 100644 index 000000000..b2c811e0b --- /dev/null +++ b/app/Exceptions/ZipExportException.php @@ -0,0 +1,7 @@ +data['page'] = [ + 'id' => $page->id, + ]; + + return $this->build(); + } + + /** + * @throws ZipExportException + */ + protected function build(): string + { + $this->data['exported_at'] = date(DATE_ATOM); + $this->data['instance'] = [ + 'version' => trim(file_get_contents(base_path('version'))), + 'id_ciphertext' => encrypt('bookstack'), + ]; + + $zipFile = tempnam(sys_get_temp_dir(), 'bszip-'); + $zip = new ZipArchive(); + $opened = $zip->open($zipFile, ZipArchive::CREATE); + if ($opened !== true) { + throw new ZipExportException('Failed to create zip file for export.'); + } + + $zip->addFromString('data.json', json_encode($this->data)); + $zip->addEmptyDir('files'); + + return $zipFile; + } +} diff --git a/composer.json b/composer.json index 5c54774f1..3680a2c6a 100644 --- a/composer.json +++ b/composer.json @@ -16,6 +16,7 @@ "ext-json": "*", "ext-mbstring": "*", "ext-xml": "*", + "ext-zip": "*", "bacon/bacon-qr-code": "^3.0", "doctrine/dbal": "^3.5", "dompdf/dompdf": "^3.0", diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index dc21bf8e5..d5635bd39 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -39,18 +39,24 @@ Some properties in the export data JSON are indicated as `String reference`, and } ``` -TODO - Jotting out idea below. -Would need to validate image/attachment paths against image/attachments listed across all pages in export. -Probably good to ensure filenames are ascii-alpha-num. -`[[bsexport:image:an-image-path.png]]` -`[[bsexport:attachment:an-image-path.png]]` -`[[bsexport:page:1]]` -`[[bsexport:chapter:2]]` -`[[bsexport:book:3]]` +Within HTML and markdown content, you may require references across to other items within the export content. +This can be done using the following format: -TODO - Define how we reference across content: -TODO - References from in-content to file URLs -TODO - References from in-content to in-export content (page cross links within same export). +``` +[[bsexport::]] +``` + +Images and attachments are referenced via their file name within the `files/` directory. +Otherwise, other content types are referenced by `id`. +Here's an example of each type of such reference that could be used: + +``` +[[bsexport:image:an-image-path.png]] +[[bsexport:attachment:an-image-path.png]] +[[bsexport:page:40]] +[[bsexport:chapter:2]] +[[bsexport:book:8]] +``` ## Export Data - `data.json` diff --git a/routes/api.php b/routes/api.php index c0919d324..710364855 100644 --- a/routes/api.php +++ b/routes/api.php @@ -9,6 +9,7 @@ use BookStack\Activity\Controllers\AuditLogApiController; use BookStack\Api\ApiDocsController; use BookStack\Entities\Controllers as EntityControllers; +use BookStack\Exports\Controllers as ExportControllers; use BookStack\Permissions\ContentPermissionApiController; use BookStack\Search\SearchApiController; use BookStack\Uploads\Controllers\AttachmentApiController; @@ -31,21 +32,20 @@ Route::get('books/{id}', [EntityControllers\BookApiController::class, 'read']); Route::put('books/{id}', [EntityControllers\BookApiController::class, 'update']); Route::delete('books/{id}', [EntityControllers\BookApiController::class, 'delete']); -Route::get('books/{id}/export/html', [EntityControllers\BookExportApiController::class, 'exportHtml']); -Route::get('books/{id}/export/pdf', [EntityControllers\BookExportApiController::class, 'exportPdf']); -Route::get('books/{id}/export/plaintext', [EntityControllers\BookExportApiController::class, 'exportPlainText']); -Route::get('books/{id}/export/markdown', [EntityControllers\BookExportApiController::class, 'exportMarkdown']); +Route::get('books/{id}/export/html', [ExportControllers\BookExportApiController::class, 'exportHtml']); +Route::get('books/{id}/export/pdf', [ExportControllers\BookExportApiController::class, 'exportPdf']); +Route::get('books/{id}/export/plaintext', [ExportControllers\BookExportApiController::class, 'exportPlainText']); +Route::get('books/{id}/export/markdown', [ExportControllers\BookExportApiController::class, 'exportMarkdown']); Route::get('chapters', [EntityControllers\ChapterApiController::class, 'list']); Route::post('chapters', [EntityControllers\ChapterApiController::class, 'create']); Route::get('chapters/{id}', [EntityControllers\ChapterApiController::class, 'read']); Route::put('chapters/{id}', [EntityControllers\ChapterApiController::class, 'update']); Route::delete('chapters/{id}', [EntityControllers\ChapterApiController::class, 'delete']); - -Route::get('chapters/{id}/export/html', [EntityControllers\ChapterExportApiController::class, 'exportHtml']); -Route::get('chapters/{id}/export/pdf', [EntityControllers\ChapterExportApiController::class, 'exportPdf']); -Route::get('chapters/{id}/export/plaintext', [EntityControllers\ChapterExportApiController::class, 'exportPlainText']); -Route::get('chapters/{id}/export/markdown', [EntityControllers\ChapterExportApiController::class, 'exportMarkdown']); +Route::get('chapters/{id}/export/html', [ExportControllers\ChapterExportApiController::class, 'exportHtml']); +Route::get('chapters/{id}/export/pdf', [ExportControllers\ChapterExportApiController::class, 'exportPdf']); +Route::get('chapters/{id}/export/plaintext', [ExportControllers\ChapterExportApiController::class, 'exportPlainText']); +Route::get('chapters/{id}/export/markdown', [ExportControllers\ChapterExportApiController::class, 'exportMarkdown']); Route::get('pages', [EntityControllers\PageApiController::class, 'list']); Route::post('pages', [EntityControllers\PageApiController::class, 'create']); @@ -53,10 +53,10 @@ Route::get('pages/{id}', [EntityControllers\PageApiController::class, 'read']); Route::put('pages/{id}', [EntityControllers\PageApiController::class, 'update']); Route::delete('pages/{id}', [EntityControllers\PageApiController::class, 'delete']); -Route::get('pages/{id}/export/html', [EntityControllers\PageExportApiController::class, 'exportHtml']); -Route::get('pages/{id}/export/pdf', [EntityControllers\PageExportApiController::class, 'exportPdf']); -Route::get('pages/{id}/export/plaintext', [EntityControllers\PageExportApiController::class, 'exportPlainText']); -Route::get('pages/{id}/export/markdown', [EntityControllers\PageExportApiController::class, 'exportMarkdown']); +Route::get('pages/{id}/export/html', [ExportControllers\PageExportApiController::class, 'exportHtml']); +Route::get('pages/{id}/export/pdf', [ExportControllers\PageExportApiController::class, 'exportPdf']); +Route::get('pages/{id}/export/plaintext', [ExportControllers\PageExportApiController::class, 'exportPlainText']); +Route::get('pages/{id}/export/markdown', [ExportControllers\PageExportApiController::class, 'exportMarkdown']); Route::get('image-gallery', [ImageGalleryApiController::class, 'list']); Route::post('image-gallery', [ImageGalleryApiController::class, 'create']); diff --git a/routes/web.php b/routes/web.php index 81b938f32..5220684c0 100644 --- a/routes/web.php +++ b/routes/web.php @@ -7,6 +7,7 @@ use BookStack\Api\UserApiTokenController; use BookStack\App\HomeController; use BookStack\App\MetaController; use BookStack\Entities\Controllers as EntityControllers; +use BookStack\Exports\Controllers as ExportControllers; use BookStack\Http\Middleware\VerifyCsrfToken; use BookStack\Permissions\PermissionsController; use BookStack\References\ReferenceController; @@ -74,11 +75,11 @@ Route::middleware('auth')->group(function () { Route::get('/books/{bookSlug}/sort', [EntityControllers\BookSortController::class, 'show']); Route::put('/books/{bookSlug}/sort', [EntityControllers\BookSortController::class, 'update']); Route::get('/books/{slug}/references', [ReferenceController::class, 'book']); - Route::get('/books/{bookSlug}/export/html', [EntityControllers\BookExportController::class, 'html']); - Route::get('/books/{bookSlug}/export/pdf', [EntityControllers\BookExportController::class, 'pdf']); - Route::get('/books/{bookSlug}/export/markdown', [EntityControllers\BookExportController::class, 'markdown']); - Route::get('/books/{bookSlug}/export/zip', [EntityControllers\BookExportController::class, 'zip']); - Route::get('/books/{bookSlug}/export/plaintext', [EntityControllers\BookExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/export/html', [ExportControllers\BookExportController::class, 'html']); + Route::get('/books/{bookSlug}/export/pdf', [ExportControllers\BookExportController::class, 'pdf']); + Route::get('/books/{bookSlug}/export/markdown', [ExportControllers\BookExportController::class, 'markdown']); + Route::get('/books/{bookSlug}/export/zip', [ExportControllers\BookExportController::class, 'zip']); + Route::get('/books/{bookSlug}/export/plaintext', [ExportControllers\BookExportController::class, 'plainText']); // Pages Route::get('/books/{bookSlug}/create-page', [EntityControllers\PageController::class, 'create']); @@ -86,10 +87,10 @@ Route::middleware('auth')->group(function () { Route::get('/books/{bookSlug}/draft/{pageId}', [EntityControllers\PageController::class, 'editDraft']); Route::post('/books/{bookSlug}/draft/{pageId}', [EntityControllers\PageController::class, 'store']); Route::get('/books/{bookSlug}/page/{pageSlug}', [EntityControllers\PageController::class, 'show']); - Route::get('/books/{bookSlug}/page/{pageSlug}/export/pdf', [EntityControllers\PageExportController::class, 'pdf']); - Route::get('/books/{bookSlug}/page/{pageSlug}/export/html', [EntityControllers\PageExportController::class, 'html']); - Route::get('/books/{bookSlug}/page/{pageSlug}/export/markdown', [EntityControllers\PageExportController::class, 'markdown']); - Route::get('/books/{bookSlug}/page/{pageSlug}/export/plaintext', [EntityControllers\PageExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/pdf', [ExportControllers\PageExportController::class, 'pdf']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/html', [ExportControllers\PageExportController::class, 'html']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/markdown', [ExportControllers\PageExportController::class, 'markdown']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/plaintext', [ExportControllers\PageExportController::class, 'plainText']); Route::get('/books/{bookSlug}/page/{pageSlug}/edit', [EntityControllers\PageController::class, 'edit']); Route::get('/books/{bookSlug}/page/{pageSlug}/move', [EntityControllers\PageController::class, 'showMove']); Route::put('/books/{bookSlug}/page/{pageSlug}/move', [EntityControllers\PageController::class, 'move']); @@ -126,10 +127,10 @@ Route::middleware('auth')->group(function () { Route::get('/books/{bookSlug}/chapter/{chapterSlug}/edit', [EntityControllers\ChapterController::class, 'edit']); Route::post('/books/{bookSlug}/chapter/{chapterSlug}/convert-to-book', [EntityControllers\ChapterController::class, 'convertToBook']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/permissions', [PermissionsController::class, 'showForChapter']); - Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/pdf', [EntityControllers\ChapterExportController::class, 'pdf']); - Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/html', [EntityControllers\ChapterExportController::class, 'html']); - Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/markdown', [EntityControllers\ChapterExportController::class, 'markdown']); - Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/plaintext', [EntityControllers\ChapterExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/pdf', [ExportControllers\ChapterExportController::class, 'pdf']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/html', [ExportControllers\ChapterExportController::class, 'html']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/markdown', [ExportControllers\ChapterExportController::class, 'markdown']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/plaintext', [ExportControllers\ChapterExportController::class, 'plainText']); Route::put('/books/{bookSlug}/chapter/{chapterSlug}/permissions', [PermissionsController::class, 'updateForChapter']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/references', [ReferenceController::class, 'chapter']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/delete', [EntityControllers\ChapterController::class, 'showDelete']); diff --git a/tests/Entity/ExportTest.php b/tests/Entity/ExportTest.php index 7aafa3b79..11cfddb20 100644 --- a/tests/Entity/ExportTest.php +++ b/tests/Entity/ExportTest.php @@ -5,8 +5,8 @@ namespace Tests\Entity; use BookStack\Entities\Models\Book; use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; -use BookStack\Entities\Tools\PdfGenerator; use BookStack\Exceptions\PdfExportException; +use BookStack\Exports\PdfGenerator; use Illuminate\Support\Facades\Storage; use Tests\TestCase; From bf0262d7d178b494e256ff5164a8d75195cdc231 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 19 Oct 2024 13:59:42 +0100 Subject: [PATCH 05/41] Testing: Split export tests into multiple files --- tests/Entity/ExportTest.php | 569 --------------------------- tests/Exports/ExportUiTest.php | 33 ++ tests/Exports/HtmlExportTest.php | 253 ++++++++++++ tests/Exports/MarkdownExportTest.php | 85 ++++ tests/Exports/PdfExportTest.php | 146 +++++++ tests/Exports/TextExportTest.php | 88 +++++ 6 files changed, 605 insertions(+), 569 deletions(-) delete mode 100644 tests/Entity/ExportTest.php create mode 100644 tests/Exports/ExportUiTest.php create mode 100644 tests/Exports/HtmlExportTest.php create mode 100644 tests/Exports/MarkdownExportTest.php create mode 100644 tests/Exports/PdfExportTest.php create mode 100644 tests/Exports/TextExportTest.php diff --git a/tests/Entity/ExportTest.php b/tests/Entity/ExportTest.php deleted file mode 100644 index 11cfddb20..000000000 --- a/tests/Entity/ExportTest.php +++ /dev/null @@ -1,569 +0,0 @@ -entities->page(); - $this->asEditor(); - - $resp = $this->get($page->getUrl('/export/plaintext')); - $resp->assertStatus(200); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.txt"'); - } - - public function test_page_pdf_export() - { - $page = $this->entities->page(); - $this->asEditor(); - - $resp = $this->get($page->getUrl('/export/pdf')); - $resp->assertStatus(200); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.pdf"'); - } - - public function test_page_html_export() - { - $page = $this->entities->page(); - $this->asEditor(); - - $resp = $this->get($page->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.html"'); - } - - public function test_book_text_export() - { - $book = $this->entities->bookHasChaptersAndPages(); - $directPage = $book->directPages()->first(); - $chapter = $book->chapters()->first(); - $chapterPage = $chapter->pages()->first(); - $this->entities->updatePage($directPage, ['html' => '

My awesome page

']); - $this->entities->updatePage($chapterPage, ['html' => '

My little nested page

']); - $this->asEditor(); - - $resp = $this->get($book->getUrl('/export/plaintext')); - $resp->assertStatus(200); - $resp->assertSee($book->name); - $resp->assertSee($chapterPage->name); - $resp->assertSee($chapter->name); - $resp->assertSee($directPage->name); - $resp->assertSee('My awesome page'); - $resp->assertSee('My little nested page'); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.txt"'); - } - - public function test_book_text_export_format() - { - $entities = $this->entities->createChainBelongingToUser($this->users->viewer()); - $this->entities->updatePage($entities['page'], ['html' => '

My great page

Full of great stuff

', 'name' => 'My wonderful page!']); - $entities['chapter']->name = 'Export chapter'; - $entities['chapter']->description = "A test chapter to be exported\nIt has loads of info within"; - $entities['book']->name = 'Export Book'; - $entities['book']->description = "This is a book with stuff to export"; - $entities['chapter']->save(); - $entities['book']->save(); - - $resp = $this->asEditor()->get($entities['book']->getUrl('/export/plaintext')); - - $expected = "Export Book\nThis is a book with stuff to export\n\nExport chapter\nA test chapter to be exported\nIt has loads of info within\n\n"; - $expected .= "My wonderful page!\nMy great page Full of great stuff"; - $resp->assertSee($expected); - } - - public function test_book_pdf_export() - { - $page = $this->entities->page(); - $book = $page->book; - $this->asEditor(); - - $resp = $this->get($book->getUrl('/export/pdf')); - $resp->assertStatus(200); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.pdf"'); - } - - public function test_book_html_export() - { - $page = $this->entities->page(); - $book = $page->book; - $this->asEditor(); - - $resp = $this->get($book->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertSee($book->name); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.html"'); - } - - public function test_book_html_export_shows_html_descriptions() - { - $book = $this->entities->bookHasChaptersAndPages(); - $chapter = $book->chapters()->first(); - $book->description_html = '

A description with HTML within!

'; - $chapter->description_html = '

A chapter description with HTML within!

'; - $book->save(); - $chapter->save(); - - $resp = $this->asEditor()->get($book->getUrl('/export/html')); - $resp->assertSee($book->description_html, false); - $resp->assertSee($chapter->description_html, false); - } - - public function test_chapter_text_export() - { - $chapter = $this->entities->chapter(); - $page = $chapter->pages[0]; - $this->entities->updatePage($page, ['html' => '

This is content within the page!

']); - $this->asEditor(); - - $resp = $this->get($chapter->getUrl('/export/plaintext')); - $resp->assertStatus(200); - $resp->assertSee($chapter->name); - $resp->assertSee($page->name); - $resp->assertSee('This is content within the page!'); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.txt"'); - } - - public function test_chapter_text_export_format() - { - $entities = $this->entities->createChainBelongingToUser($this->users->viewer()); - $this->entities->updatePage($entities['page'], ['html' => '

My great page

Full of great stuff

', 'name' => 'My wonderful page!']); - $entities['chapter']->name = 'Export chapter'; - $entities['chapter']->description = "A test chapter to be exported\nIt has loads of info within"; - $entities['chapter']->save(); - - $resp = $this->asEditor()->get($entities['book']->getUrl('/export/plaintext')); - - $expected = "Export chapter\nA test chapter to be exported\nIt has loads of info within\n\n"; - $expected .= "My wonderful page!\nMy great page Full of great stuff"; - $resp->assertSee($expected); - } - - public function test_chapter_pdf_export() - { - $chapter = $this->entities->chapter(); - $this->asEditor(); - - $resp = $this->get($chapter->getUrl('/export/pdf')); - $resp->assertStatus(200); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.pdf"'); - } - - public function test_chapter_html_export() - { - $chapter = $this->entities->chapter(); - $page = $chapter->pages[0]; - $this->asEditor(); - - $resp = $this->get($chapter->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertSee($chapter->name); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.html"'); - } - - public function test_chapter_html_export_shows_html_descriptions() - { - $chapter = $this->entities->chapter(); - $chapter->description_html = '

A description with HTML within!

'; - $chapter->save(); - - $resp = $this->asEditor()->get($chapter->getUrl('/export/html')); - $resp->assertSee($chapter->description_html, false); - } - - public function test_page_html_export_contains_custom_head_if_set() - { - $page = $this->entities->page(); - - $customHeadContent = ''; - $this->setSettings(['app-custom-head' => $customHeadContent]); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertSee($customHeadContent, false); - } - - public function test_page_html_export_does_not_break_with_only_comments_in_custom_head() - { - $page = $this->entities->page(); - - $customHeadContent = ''; - $this->setSettings(['app-custom-head' => $customHeadContent]); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertSee($customHeadContent, false); - } - - public function test_page_html_export_use_absolute_dates() - { - $page = $this->entities->page(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertSee($page->created_at->isoFormat('D MMMM Y HH:mm:ss')); - $resp->assertDontSee($page->created_at->diffForHumans()); - $resp->assertSee($page->updated_at->isoFormat('D MMMM Y HH:mm:ss')); - $resp->assertDontSee($page->updated_at->diffForHumans()); - } - - public function test_page_export_does_not_include_user_or_revision_links() - { - $page = $this->entities->page(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertDontSee($page->getUrl('/revisions')); - $resp->assertDontSee($page->createdBy->getProfileUrl()); - $resp->assertSee($page->createdBy->name); - } - - public function test_page_export_sets_right_data_type_for_svg_embeds() - { - $page = $this->entities->page(); - Storage::disk('local')->makeDirectory('uploads/images/gallery'); - Storage::disk('local')->put('uploads/images/gallery/svg_test.svg', ''); - $page->html = ''; - $page->save(); - - $this->asEditor(); - $resp = $this->get($page->getUrl('/export/html')); - Storage::disk('local')->delete('uploads/images/gallery/svg_test.svg'); - - $resp->assertStatus(200); - $resp->assertSee(''; - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - Storage::disk('local')->delete('uploads/images/gallery/svg_test.svg'); - Storage::disk('local')->delete('uploads/images/gallery/svg_test2.svg'); - - $resp->assertDontSee('http://localhost/uploads/images/gallery/svg_test'); - } - - public function test_page_export_contained_html_image_fetches_only_run_when_url_points_to_image_upload_folder() - { - $page = $this->entities->page(); - $page->html = '' - . '' - . ''; - $storageDisk = Storage::disk('local'); - $storageDisk->makeDirectory('uploads/images/gallery'); - $storageDisk->put('uploads/images/gallery/svg_test.svg', 'good'); - $storageDisk->put('uploads/svg_test.svg', 'bad'); - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - - $storageDisk->delete('uploads/images/gallery/svg_test.svg'); - $storageDisk->delete('uploads/svg_test.svg'); - - $resp->assertDontSee('http://localhost/uploads/images/gallery/svg_test.svg', false); - $resp->assertSee('http://localhost/uploads/svg_test.svg'); - $resp->assertSee('src="/uploads/svg_test.svg"', false); - } - - public function test_page_export_contained_html_does_not_allow_upward_traversal_with_local() - { - $contents = file_get_contents(public_path('.htaccess')); - config()->set('filesystems.images', 'local'); - - $page = $this->entities->page(); - $page->html = ''; - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertDontSee(base64_encode($contents)); - } - - public function test_page_export_contained_html_does_not_allow_upward_traversal_with_local_secure() - { - $testFilePath = storage_path('logs/test.txt'); - config()->set('filesystems.images', 'local_secure'); - file_put_contents($testFilePath, 'I am a cat'); - - $page = $this->entities->page(); - $page->html = ''; - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertDontSee(base64_encode('I am a cat')); - unlink($testFilePath); - } - - public function test_exports_removes_scripts_from_custom_head() - { - $entities = [ - Page::query()->first(), Chapter::query()->first(), Book::query()->first(), - ]; - setting()->put('app-custom-head', ''); - - foreach ($entities as $entity) { - $resp = $this->asEditor()->get($entity->getUrl('/export/html')); - $resp->assertDontSee('window.donkey'); - $resp->assertDontSee('assertSee('.my-test-class { color: red; }'); - } - } - - public function test_page_export_with_deleted_creator_and_updater() - { - $user = $this->users->viewer(['name' => 'ExportWizardTheFifth']); - $page = $this->entities->page(); - $page->created_by = $user->id; - $page->updated_by = $user->id; - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $resp->assertSee('ExportWizardTheFifth'); - - $user->delete(); - $resp = $this->get($page->getUrl('/export/html')); - $resp->assertStatus(200); - $resp->assertDontSee('ExportWizardTheFifth'); - } - - public function test_page_pdf_export_converts_iframes_to_links() - { - $page = Page::query()->first()->forceFill([ - 'html' => '', - ]); - $page->save(); - - $pdfHtml = ''; - $mockPdfGenerator = $this->mock(PdfGenerator::class); - $mockPdfGenerator->shouldReceive('fromHtml') - ->with(\Mockery::capture($pdfHtml)) - ->andReturn(''); - $mockPdfGenerator->shouldReceive('getActiveEngine')->andReturn(PdfGenerator::ENGINE_DOMPDF); - - $this->asEditor()->get($page->getUrl('/export/pdf')); - $this->assertStringNotContainsString('iframe>', $pdfHtml); - $this->assertStringContainsString('

https://www.youtube.com/embed/ShqUjt33uOs

', $pdfHtml); - } - - public function test_page_pdf_export_opens_details_blocks() - { - $page = $this->entities->page()->forceFill([ - 'html' => '
Hello

Content!

', - ]); - $page->save(); - - $pdfHtml = ''; - $mockPdfGenerator = $this->mock(PdfGenerator::class); - $mockPdfGenerator->shouldReceive('fromHtml') - ->with(\Mockery::capture($pdfHtml)) - ->andReturn(''); - $mockPdfGenerator->shouldReceive('getActiveEngine')->andReturn(PdfGenerator::ENGINE_DOMPDF); - - $this->asEditor()->get($page->getUrl('/export/pdf')); - $this->assertStringContainsString('
entities->page(); - - $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); - $resp->assertStatus(200); - $resp->assertSee($page->name); - $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.md"'); - } - - public function test_page_markdown_export_uses_existing_markdown_if_apparent() - { - $page = $this->entities->page()->forceFill([ - 'markdown' => '# A header', - 'html' => '

Dogcat

', - ]); - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); - $resp->assertSee('A header'); - $resp->assertDontSee('Dogcat'); - } - - public function test_page_markdown_export_converts_html_where_no_markdown() - { - $page = $this->entities->page()->forceFill([ - 'markdown' => '', - 'html' => '

Dogcat

Some bold text

', - ]); - $page->save(); - - $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); - $resp->assertSee("# Dogcat\n\nSome **bold** text"); - } - - public function test_chapter_markdown_export() - { - $chapter = $this->entities->chapter(); - $page = $chapter->pages()->first(); - $resp = $this->asEditor()->get($chapter->getUrl('/export/markdown')); - - $resp->assertSee('# ' . $chapter->name); - $resp->assertSee('# ' . $page->name); - } - - public function test_book_markdown_export() - { - $book = Book::query()->whereHas('pages')->whereHas('chapters')->first(); - $chapter = $book->chapters()->first(); - $page = $chapter->pages()->first(); - $resp = $this->asEditor()->get($book->getUrl('/export/markdown')); - - $resp->assertSee('# ' . $book->name); - $resp->assertSee('# ' . $chapter->name); - $resp->assertSee('# ' . $page->name); - } - - public function test_book_markdown_export_concats_immediate_pages_with_newlines() - { - /** @var Book $book */ - $book = Book::query()->whereHas('pages')->first(); - - $this->asEditor()->get($book->getUrl('/create-page')); - $this->get($book->getUrl('/create-page')); - - [$pageA, $pageB] = $book->pages()->where('chapter_id', '=', 0)->get(); - $pageA->html = '

hello tester

'; - $pageA->save(); - $pageB->name = 'The second page in this test'; - $pageB->save(); - - $resp = $this->get($book->getUrl('/export/markdown')); - $resp->assertDontSee('hello tester# The second page in this test'); - $resp->assertSee("hello tester\n\n# The second page in this test"); - } - - public function test_export_option_only_visible_and_accessible_with_permission() - { - $book = Book::query()->whereHas('pages')->whereHas('chapters')->first(); - $chapter = $book->chapters()->first(); - $page = $chapter->pages()->first(); - $entities = [$book, $chapter, $page]; - $user = $this->users->viewer(); - $this->actingAs($user); - - foreach ($entities as $entity) { - $resp = $this->get($entity->getUrl()); - $resp->assertSee('/export/pdf'); - } - - $this->permissions->removeUserRolePermissions($user, ['content-export']); - - foreach ($entities as $entity) { - $resp = $this->get($entity->getUrl()); - $resp->assertDontSee('/export/pdf'); - $resp = $this->get($entity->getUrl('/export/pdf')); - $this->assertPermissionError($resp); - } - } - - public function test_wkhtmltopdf_only_used_when_allow_untrusted_is_true() - { - $page = $this->entities->page(); - - config()->set('exports.snappy.pdf_binary', '/abc123'); - config()->set('app.allow_untrusted_server_fetching', false); - - $resp = $this->asEditor()->get($page->getUrl('/export/pdf')); - $resp->assertStatus(200); // Sucessful response with invalid snappy binary indicates dompdf usage. - - config()->set('app.allow_untrusted_server_fetching', true); - $resp = $this->get($page->getUrl('/export/pdf')); - $resp->assertStatus(500); // Bad response indicates wkhtml usage - } - - public function test_pdf_command_option_used_if_set() - { - $page = $this->entities->page(); - $command = 'cp {input_html_path} {output_pdf_path}'; - config()->set('exports.pdf_command', $command); - - $resp = $this->asEditor()->get($page->getUrl('/export/pdf')); - $download = $resp->getContent(); - - $this->assertStringContainsString(e($page->name), $download); - $this->assertStringContainsString('set('exports.pdf_command', $command); - - $this->assertThrows(function () use ($page) { - $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); - }, PdfExportException::class); - } - - public function test_pdf_command_option_errors_if_command_returns_error_status() - { - $page = $this->entities->page(); - $command = 'exit 1'; - config()->set('exports.pdf_command', $command); - - $this->assertThrows(function () use ($page) { - $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); - }, PdfExportException::class); - } - - public function test_pdf_command_timout_option_limits_export_time() - { - $page = $this->entities->page(); - $command = 'php -r \'sleep(4);\''; - config()->set('exports.pdf_command', $command); - config()->set('exports.pdf_command_timeout', 1); - - $this->assertThrows(function () use ($page) { - $start = time(); - $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); - - $this->assertTrue(time() < ($start + 3)); - }, PdfExportException::class, - "PDF Export via command failed due to timeout at 1 second(s)"); - } - - public function test_html_exports_contain_csp_meta_tag() - { - $entities = [ - $this->entities->page(), - $this->entities->book(), - $this->entities->chapter(), - ]; - - foreach ($entities as $entity) { - $resp = $this->asEditor()->get($entity->getUrl('/export/html')); - $this->withHtml($resp)->assertElementExists('head meta[http-equiv="Content-Security-Policy"][content*="script-src "]'); - } - } - - public function test_html_exports_contain_body_classes_for_export_identification() - { - $page = $this->entities->page(); - - $resp = $this->asEditor()->get($page->getUrl('/export/html')); - $this->withHtml($resp)->assertElementExists('body.export.export-format-html.export-engine-none'); - } -} diff --git a/tests/Exports/ExportUiTest.php b/tests/Exports/ExportUiTest.php new file mode 100644 index 000000000..77b26ad89 --- /dev/null +++ b/tests/Exports/ExportUiTest.php @@ -0,0 +1,33 @@ +whereHas('pages')->whereHas('chapters')->first(); + $chapter = $book->chapters()->first(); + $page = $chapter->pages()->first(); + $entities = [$book, $chapter, $page]; + $user = $this->users->viewer(); + $this->actingAs($user); + + foreach ($entities as $entity) { + $resp = $this->get($entity->getUrl()); + $resp->assertSee('/export/pdf'); + } + + $this->permissions->removeUserRolePermissions($user, ['content-export']); + + foreach ($entities as $entity) { + $resp = $this->get($entity->getUrl()); + $resp->assertDontSee('/export/pdf'); + $resp = $this->get($entity->getUrl('/export/pdf')); + $this->assertPermissionError($resp); + } + } +} diff --git a/tests/Exports/HtmlExportTest.php b/tests/Exports/HtmlExportTest.php new file mode 100644 index 000000000..069cf2801 --- /dev/null +++ b/tests/Exports/HtmlExportTest.php @@ -0,0 +1,253 @@ +entities->page(); + $this->asEditor(); + + $resp = $this->get($page->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.html"'); + } + + public function test_book_html_export() + { + $page = $this->entities->page(); + $book = $page->book; + $this->asEditor(); + + $resp = $this->get($book->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertSee($book->name); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.html"'); + } + + public function test_book_html_export_shows_html_descriptions() + { + $book = $this->entities->bookHasChaptersAndPages(); + $chapter = $book->chapters()->first(); + $book->description_html = '

A description with HTML within!

'; + $chapter->description_html = '

A chapter description with HTML within!

'; + $book->save(); + $chapter->save(); + + $resp = $this->asEditor()->get($book->getUrl('/export/html')); + $resp->assertSee($book->description_html, false); + $resp->assertSee($chapter->description_html, false); + } + + public function test_chapter_html_export() + { + $chapter = $this->entities->chapter(); + $page = $chapter->pages[0]; + $this->asEditor(); + + $resp = $this->get($chapter->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertSee($chapter->name); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.html"'); + } + + public function test_chapter_html_export_shows_html_descriptions() + { + $chapter = $this->entities->chapter(); + $chapter->description_html = '

A description with HTML within!

'; + $chapter->save(); + + $resp = $this->asEditor()->get($chapter->getUrl('/export/html')); + $resp->assertSee($chapter->description_html, false); + } + + public function test_page_html_export_contains_custom_head_if_set() + { + $page = $this->entities->page(); + + $customHeadContent = ''; + $this->setSettings(['app-custom-head' => $customHeadContent]); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertSee($customHeadContent, false); + } + + public function test_page_html_export_does_not_break_with_only_comments_in_custom_head() + { + $page = $this->entities->page(); + + $customHeadContent = ''; + $this->setSettings(['app-custom-head' => $customHeadContent]); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertSee($customHeadContent, false); + } + + public function test_page_html_export_use_absolute_dates() + { + $page = $this->entities->page(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertSee($page->created_at->isoFormat('D MMMM Y HH:mm:ss')); + $resp->assertDontSee($page->created_at->diffForHumans()); + $resp->assertSee($page->updated_at->isoFormat('D MMMM Y HH:mm:ss')); + $resp->assertDontSee($page->updated_at->diffForHumans()); + } + + public function test_page_export_does_not_include_user_or_revision_links() + { + $page = $this->entities->page(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertDontSee($page->getUrl('/revisions')); + $resp->assertDontSee($page->createdBy->getProfileUrl()); + $resp->assertSee($page->createdBy->name); + } + + public function test_page_export_sets_right_data_type_for_svg_embeds() + { + $page = $this->entities->page(); + Storage::disk('local')->makeDirectory('uploads/images/gallery'); + Storage::disk('local')->put('uploads/images/gallery/svg_test.svg', ''); + $page->html = ''; + $page->save(); + + $this->asEditor(); + $resp = $this->get($page->getUrl('/export/html')); + Storage::disk('local')->delete('uploads/images/gallery/svg_test.svg'); + + $resp->assertStatus(200); + $resp->assertSee(''; + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + Storage::disk('local')->delete('uploads/images/gallery/svg_test.svg'); + Storage::disk('local')->delete('uploads/images/gallery/svg_test2.svg'); + + $resp->assertDontSee('http://localhost/uploads/images/gallery/svg_test'); + } + + public function test_page_export_contained_html_image_fetches_only_run_when_url_points_to_image_upload_folder() + { + $page = $this->entities->page(); + $page->html = '' + . '' + . ''; + $storageDisk = Storage::disk('local'); + $storageDisk->makeDirectory('uploads/images/gallery'); + $storageDisk->put('uploads/images/gallery/svg_test.svg', 'good'); + $storageDisk->put('uploads/svg_test.svg', 'bad'); + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + + $storageDisk->delete('uploads/images/gallery/svg_test.svg'); + $storageDisk->delete('uploads/svg_test.svg'); + + $resp->assertDontSee('http://localhost/uploads/images/gallery/svg_test.svg', false); + $resp->assertSee('http://localhost/uploads/svg_test.svg'); + $resp->assertSee('src="/uploads/svg_test.svg"', false); + } + + public function test_page_export_contained_html_does_not_allow_upward_traversal_with_local() + { + $contents = file_get_contents(public_path('.htaccess')); + config()->set('filesystems.images', 'local'); + + $page = $this->entities->page(); + $page->html = ''; + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertDontSee(base64_encode($contents)); + } + + public function test_page_export_contained_html_does_not_allow_upward_traversal_with_local_secure() + { + $testFilePath = storage_path('logs/test.txt'); + config()->set('filesystems.images', 'local_secure'); + file_put_contents($testFilePath, 'I am a cat'); + + $page = $this->entities->page(); + $page->html = ''; + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertDontSee(base64_encode('I am a cat')); + unlink($testFilePath); + } + + public function test_exports_removes_scripts_from_custom_head() + { + $entities = [ + Page::query()->first(), Chapter::query()->first(), Book::query()->first(), + ]; + setting()->put('app-custom-head', ''); + + foreach ($entities as $entity) { + $resp = $this->asEditor()->get($entity->getUrl('/export/html')); + $resp->assertDontSee('window.donkey'); + $resp->assertDontSee('assertSee('.my-test-class { color: red; }'); + } + } + + public function test_page_export_with_deleted_creator_and_updater() + { + $user = $this->users->viewer(['name' => 'ExportWizardTheFifth']); + $page = $this->entities->page(); + $page->created_by = $user->id; + $page->updated_by = $user->id; + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $resp->assertSee('ExportWizardTheFifth'); + + $user->delete(); + $resp = $this->get($page->getUrl('/export/html')); + $resp->assertStatus(200); + $resp->assertDontSee('ExportWizardTheFifth'); + } + + public function test_html_exports_contain_csp_meta_tag() + { + $entities = [ + $this->entities->page(), + $this->entities->book(), + $this->entities->chapter(), + ]; + + foreach ($entities as $entity) { + $resp = $this->asEditor()->get($entity->getUrl('/export/html')); + $this->withHtml($resp)->assertElementExists('head meta[http-equiv="Content-Security-Policy"][content*="script-src "]'); + } + } + + public function test_html_exports_contain_body_classes_for_export_identification() + { + $page = $this->entities->page(); + + $resp = $this->asEditor()->get($page->getUrl('/export/html')); + $this->withHtml($resp)->assertElementExists('body.export.export-format-html.export-engine-none'); + } +} diff --git a/tests/Exports/MarkdownExportTest.php b/tests/Exports/MarkdownExportTest.php new file mode 100644 index 000000000..05ebbc68d --- /dev/null +++ b/tests/Exports/MarkdownExportTest.php @@ -0,0 +1,85 @@ +entities->page(); + + $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); + $resp->assertStatus(200); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.md"'); + } + + public function test_page_markdown_export_uses_existing_markdown_if_apparent() + { + $page = $this->entities->page()->forceFill([ + 'markdown' => '# A header', + 'html' => '

Dogcat

', + ]); + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); + $resp->assertSee('A header'); + $resp->assertDontSee('Dogcat'); + } + + public function test_page_markdown_export_converts_html_where_no_markdown() + { + $page = $this->entities->page()->forceFill([ + 'markdown' => '', + 'html' => '

Dogcat

Some bold text

', + ]); + $page->save(); + + $resp = $this->asEditor()->get($page->getUrl('/export/markdown')); + $resp->assertSee("# Dogcat\n\nSome **bold** text"); + } + + public function test_chapter_markdown_export() + { + $chapter = $this->entities->chapter(); + $page = $chapter->pages()->first(); + $resp = $this->asEditor()->get($chapter->getUrl('/export/markdown')); + + $resp->assertSee('# ' . $chapter->name); + $resp->assertSee('# ' . $page->name); + } + + public function test_book_markdown_export() + { + $book = Book::query()->whereHas('pages')->whereHas('chapters')->first(); + $chapter = $book->chapters()->first(); + $page = $chapter->pages()->first(); + $resp = $this->asEditor()->get($book->getUrl('/export/markdown')); + + $resp->assertSee('# ' . $book->name); + $resp->assertSee('# ' . $chapter->name); + $resp->assertSee('# ' . $page->name); + } + + public function test_book_markdown_export_concats_immediate_pages_with_newlines() + { + /** @var Book $book */ + $book = Book::query()->whereHas('pages')->first(); + + $this->asEditor()->get($book->getUrl('/create-page')); + $this->get($book->getUrl('/create-page')); + + [$pageA, $pageB] = $book->pages()->where('chapter_id', '=', 0)->get(); + $pageA->html = '

hello tester

'; + $pageA->save(); + $pageB->name = 'The second page in this test'; + $pageB->save(); + + $resp = $this->get($book->getUrl('/export/markdown')); + $resp->assertDontSee('hello tester# The second page in this test'); + $resp->assertSee("hello tester\n\n# The second page in this test"); + } +} diff --git a/tests/Exports/PdfExportTest.php b/tests/Exports/PdfExportTest.php new file mode 100644 index 000000000..9d85c69e2 --- /dev/null +++ b/tests/Exports/PdfExportTest.php @@ -0,0 +1,146 @@ +entities->page(); + $this->asEditor(); + + $resp = $this->get($page->getUrl('/export/pdf')); + $resp->assertStatus(200); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.pdf"'); + } + + public function test_book_pdf_export() + { + $page = $this->entities->page(); + $book = $page->book; + $this->asEditor(); + + $resp = $this->get($book->getUrl('/export/pdf')); + $resp->assertStatus(200); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.pdf"'); + } + + public function test_chapter_pdf_export() + { + $chapter = $this->entities->chapter(); + $this->asEditor(); + + $resp = $this->get($chapter->getUrl('/export/pdf')); + $resp->assertStatus(200); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.pdf"'); + } + + + public function test_page_pdf_export_converts_iframes_to_links() + { + $page = Page::query()->first()->forceFill([ + 'html' => '', + ]); + $page->save(); + + $pdfHtml = ''; + $mockPdfGenerator = $this->mock(PdfGenerator::class); + $mockPdfGenerator->shouldReceive('fromHtml') + ->with(\Mockery::capture($pdfHtml)) + ->andReturn(''); + $mockPdfGenerator->shouldReceive('getActiveEngine')->andReturn(PdfGenerator::ENGINE_DOMPDF); + + $this->asEditor()->get($page->getUrl('/export/pdf')); + $this->assertStringNotContainsString('iframe>', $pdfHtml); + $this->assertStringContainsString('

https://www.youtube.com/embed/ShqUjt33uOs

', $pdfHtml); + } + + public function test_page_pdf_export_opens_details_blocks() + { + $page = $this->entities->page()->forceFill([ + 'html' => '
Hello

Content!

', + ]); + $page->save(); + + $pdfHtml = ''; + $mockPdfGenerator = $this->mock(PdfGenerator::class); + $mockPdfGenerator->shouldReceive('fromHtml') + ->with(\Mockery::capture($pdfHtml)) + ->andReturn(''); + $mockPdfGenerator->shouldReceive('getActiveEngine')->andReturn(PdfGenerator::ENGINE_DOMPDF); + + $this->asEditor()->get($page->getUrl('/export/pdf')); + $this->assertStringContainsString('
entities->page(); + + config()->set('exports.snappy.pdf_binary', '/abc123'); + config()->set('app.allow_untrusted_server_fetching', false); + + $resp = $this->asEditor()->get($page->getUrl('/export/pdf')); + $resp->assertStatus(200); // Sucessful response with invalid snappy binary indicates dompdf usage. + + config()->set('app.allow_untrusted_server_fetching', true); + $resp = $this->get($page->getUrl('/export/pdf')); + $resp->assertStatus(500); // Bad response indicates wkhtml usage + } + + public function test_pdf_command_option_used_if_set() + { + $page = $this->entities->page(); + $command = 'cp {input_html_path} {output_pdf_path}'; + config()->set('exports.pdf_command', $command); + + $resp = $this->asEditor()->get($page->getUrl('/export/pdf')); + $download = $resp->getContent(); + + $this->assertStringContainsString(e($page->name), $download); + $this->assertStringContainsString('set('exports.pdf_command', $command); + + $this->assertThrows(function () use ($page) { + $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); + }, PdfExportException::class); + } + + public function test_pdf_command_option_errors_if_command_returns_error_status() + { + $page = $this->entities->page(); + $command = 'exit 1'; + config()->set('exports.pdf_command', $command); + + $this->assertThrows(function () use ($page) { + $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); + }, PdfExportException::class); + } + + public function test_pdf_command_timout_option_limits_export_time() + { + $page = $this->entities->page(); + $command = 'php -r \'sleep(4);\''; + config()->set('exports.pdf_command', $command); + config()->set('exports.pdf_command_timeout', 1); + + $this->assertThrows(function () use ($page) { + $start = time(); + $this->withoutExceptionHandling()->asEditor()->get($page->getUrl('/export/pdf')); + + $this->assertTrue(time() < ($start + 3)); + }, PdfExportException::class, + "PDF Export via command failed due to timeout at 1 second(s)"); + } +} diff --git a/tests/Exports/TextExportTest.php b/tests/Exports/TextExportTest.php new file mode 100644 index 000000000..c593a6585 --- /dev/null +++ b/tests/Exports/TextExportTest.php @@ -0,0 +1,88 @@ +entities->page(); + $this->asEditor(); + + $resp = $this->get($page->getUrl('/export/plaintext')); + $resp->assertStatus(200); + $resp->assertSee($page->name); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $page->slug . '.txt"'); + } + + public function test_book_text_export() + { + $book = $this->entities->bookHasChaptersAndPages(); + $directPage = $book->directPages()->first(); + $chapter = $book->chapters()->first(); + $chapterPage = $chapter->pages()->first(); + $this->entities->updatePage($directPage, ['html' => '

My awesome page

']); + $this->entities->updatePage($chapterPage, ['html' => '

My little nested page

']); + $this->asEditor(); + + $resp = $this->get($book->getUrl('/export/plaintext')); + $resp->assertStatus(200); + $resp->assertSee($book->name); + $resp->assertSee($chapterPage->name); + $resp->assertSee($chapter->name); + $resp->assertSee($directPage->name); + $resp->assertSee('My awesome page'); + $resp->assertSee('My little nested page'); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $book->slug . '.txt"'); + } + + public function test_book_text_export_format() + { + $entities = $this->entities->createChainBelongingToUser($this->users->viewer()); + $this->entities->updatePage($entities['page'], ['html' => '

My great page

Full of great stuff

', 'name' => 'My wonderful page!']); + $entities['chapter']->name = 'Export chapter'; + $entities['chapter']->description = "A test chapter to be exported\nIt has loads of info within"; + $entities['book']->name = 'Export Book'; + $entities['book']->description = "This is a book with stuff to export"; + $entities['chapter']->save(); + $entities['book']->save(); + + $resp = $this->asEditor()->get($entities['book']->getUrl('/export/plaintext')); + + $expected = "Export Book\nThis is a book with stuff to export\n\nExport chapter\nA test chapter to be exported\nIt has loads of info within\n\n"; + $expected .= "My wonderful page!\nMy great page Full of great stuff"; + $resp->assertSee($expected); + } + + public function test_chapter_text_export() + { + $chapter = $this->entities->chapter(); + $page = $chapter->pages[0]; + $this->entities->updatePage($page, ['html' => '

This is content within the page!

']); + $this->asEditor(); + + $resp = $this->get($chapter->getUrl('/export/plaintext')); + $resp->assertStatus(200); + $resp->assertSee($chapter->name); + $resp->assertSee($page->name); + $resp->assertSee('This is content within the page!'); + $resp->assertHeader('Content-Disposition', 'attachment; filename="' . $chapter->slug . '.txt"'); + } + + public function test_chapter_text_export_format() + { + $entities = $this->entities->createChainBelongingToUser($this->users->viewer()); + $this->entities->updatePage($entities['page'], ['html' => '

My great page

Full of great stuff

', 'name' => 'My wonderful page!']); + $entities['chapter']->name = 'Export chapter'; + $entities['chapter']->description = "A test chapter to be exported\nIt has loads of info within"; + $entities['chapter']->save(); + + $resp = $this->asEditor()->get($entities['book']->getUrl('/export/plaintext')); + + $expected = "Export chapter\nA test chapter to be exported\nIt has loads of info within\n\n"; + $expected .= "My wonderful page!\nMy great page Full of great stuff"; + $resp->assertSee($expected); + } +} From 21ccfa97ddfe6309ff735aafbc8138cf34782563 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 19 Oct 2024 15:41:07 +0100 Subject: [PATCH 06/41] ZIP Export: Expanded page & added base attachment handling --- .../Controllers/PageExportController.php | 13 ++++ app/Exports/ZipExportBuilder.php | 66 +++++++++++++++++-- app/Exports/ZipExportFiles.php | 58 ++++++++++++++++ app/Uploads/AttachmentService.php | 11 +--- lang/en/entities.php | 1 + .../views/entities/export-menu.blade.php | 1 + routes/web.php | 1 + tests/Exports/ZipExportTest.php | 15 +++++ 8 files changed, 154 insertions(+), 12 deletions(-) create mode 100644 app/Exports/ZipExportFiles.php create mode 100644 tests/Exports/ZipExportTest.php diff --git a/app/Exports/Controllers/PageExportController.php b/app/Exports/Controllers/PageExportController.php index a4e7aae87..01611fd21 100644 --- a/app/Exports/Controllers/PageExportController.php +++ b/app/Exports/Controllers/PageExportController.php @@ -6,6 +6,7 @@ use BookStack\Entities\Queries\PageQueries; use BookStack\Entities\Tools\PageContent; use BookStack\Exceptions\NotFoundException; use BookStack\Exports\ExportFormatter; +use BookStack\Exports\ZipExportBuilder; use BookStack\Http\Controller; use Throwable; @@ -74,4 +75,16 @@ class PageExportController extends Controller return $this->download()->directly($pageText, $pageSlug . '.md'); } + + /** + * Export a page to a contained ZIP export file. + * @throws NotFoundException + */ + public function zip(string $bookSlug, string $pageSlug, ZipExportBuilder $builder) + { + $page = $this->queries->findVisibleBySlugsOrFail($bookSlug, $pageSlug); + $zip = $builder->buildForPage($page); + + return $this->download()->streamedDirectly(fopen($zip, 'r'), $pageSlug . '.zip', filesize($zip)); + } } diff --git a/app/Exports/ZipExportBuilder.php b/app/Exports/ZipExportBuilder.php index d1a7b6bd4..2b8b45d0d 100644 --- a/app/Exports/ZipExportBuilder.php +++ b/app/Exports/ZipExportBuilder.php @@ -2,24 +2,70 @@ namespace BookStack\Exports; +use BookStack\Activity\Models\Tag; use BookStack\Entities\Models\Page; use BookStack\Exceptions\ZipExportException; +use BookStack\Uploads\Attachment; use ZipArchive; class ZipExportBuilder { protected array $data = []; + public function __construct( + protected ZipExportFiles $files + ) { + } + /** * @throws ZipExportException */ public function buildForPage(Page $page): string { - $this->data['page'] = [ - 'id' => $page->id, + $this->data['page'] = $this->convertPage($page); + return $this->build(); + } + + protected function convertPage(Page $page): array + { + $tags = array_map($this->convertTag(...), $page->tags()->get()->all()); + $attachments = array_map($this->convertAttachment(...), $page->attachments()->get()->all()); + + return [ + 'id' => $page->id, + 'name' => $page->name, + 'html' => '', // TODO + 'markdown' => '', // TODO + 'priority' => $page->priority, + 'attachments' => $attachments, + 'images' => [], // TODO + 'tags' => $tags, + ]; + } + + protected function convertAttachment(Attachment $attachment): array + { + $data = [ + 'name' => $attachment->name, + 'order' => $attachment->order, ]; - return $this->build(); + if ($attachment->external) { + $data['link'] = $attachment->path; + } else { + $data['file'] = $this->files->referenceForAttachment($attachment); + } + + return $data; + } + + protected function convertTag(Tag $tag): array + { + return [ + 'name' => $tag->name, + 'value' => $tag->value, + 'order' => $tag->order, + ]; } /** @@ -29,7 +75,7 @@ class ZipExportBuilder { $this->data['exported_at'] = date(DATE_ATOM); $this->data['instance'] = [ - 'version' => trim(file_get_contents(base_path('version'))), + 'version' => trim(file_get_contents(base_path('version'))), 'id_ciphertext' => encrypt('bookstack'), ]; @@ -43,6 +89,18 @@ class ZipExportBuilder $zip->addFromString('data.json', json_encode($this->data)); $zip->addEmptyDir('files'); + $toRemove = []; + $this->files->extractEach(function ($filePath, $fileRef) use ($zip, &$toRemove) { + $zip->addFile($filePath, "files/$fileRef"); + $toRemove[] = $filePath; + }); + + $zip->close(); + + foreach ($toRemove as $file) { + unlink($file); + } + return $zipFile; } } diff --git a/app/Exports/ZipExportFiles.php b/app/Exports/ZipExportFiles.php new file mode 100644 index 000000000..d3ee70e93 --- /dev/null +++ b/app/Exports/ZipExportFiles.php @@ -0,0 +1,58 @@ + + */ + protected array $attachmentRefsById = []; + + public function __construct( + protected AttachmentService $attachmentService, + ) { + } + + /** + * Gain a reference to the given attachment instance. + * This is expected to be a file-based attachment that the user + * has visibility of, no permission/access checks are performed here. + */ + public function referenceForAttachment(Attachment $attachment): string + { + if (isset($this->attachmentRefsById[$attachment->id])) { + return $this->attachmentRefsById[$attachment->id]; + } + + do { + $fileName = Str::random(20) . '.' . $attachment->extension; + } while (in_array($fileName, $this->attachmentRefsById)); + + $this->attachmentRefsById[$attachment->id] = $fileName; + + return $fileName; + } + + /** + * Extract each of the ZIP export tracked files. + * Calls the given callback for each tracked file, passing a temporary + * file reference of the file contents, and the zip-local tracked reference. + */ + public function extractEach(callable $callback): void + { + foreach ($this->attachmentRefsById as $attachmentId => $ref) { + $attachment = Attachment::query()->find($attachmentId); + $stream = $this->attachmentService->streamAttachmentFromStorage($attachment); + $tmpFile = tempnam(sys_get_temp_dir(), 'bszipfile-'); + $tmpFileStream = fopen($tmpFile, 'w'); + stream_copy_to_stream($stream, $tmpFileStream); + $callback($tmpFile, $ref); + } + } +} diff --git a/app/Uploads/AttachmentService.php b/app/Uploads/AttachmentService.php index bd319fbd7..227649d8f 100644 --- a/app/Uploads/AttachmentService.php +++ b/app/Uploads/AttachmentService.php @@ -13,14 +13,9 @@ use Symfony\Component\HttpFoundation\File\UploadedFile; class AttachmentService { - protected FilesystemManager $fileSystem; - - /** - * AttachmentService constructor. - */ - public function __construct(FilesystemManager $fileSystem) - { - $this->fileSystem = $fileSystem; + public function __construct( + protected FilesystemManager $fileSystem + ) { } /** diff --git a/lang/en/entities.php b/lang/en/entities.php index 35e6f050b..7e5a708ef 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -39,6 +39,7 @@ return [ 'export_pdf' => 'PDF File', 'export_text' => 'Plain Text File', 'export_md' => 'Markdown File', + 'export_zip' => 'Portable ZIP', 'default_template' => 'Default Page Template', 'default_template_explain' => 'Assign a page template that will be used as the default content for all pages created within this item. Keep in mind this will only be used if the page creator has view access to the chosen template page.', 'default_template_select' => 'Select a template page', diff --git a/resources/views/entities/export-menu.blade.php b/resources/views/entities/export-menu.blade.php index a55ab56d1..e58c842ba 100644 --- a/resources/views/entities/export-menu.blade.php +++ b/resources/views/entities/export-menu.blade.php @@ -18,6 +18,7 @@
  • {{ trans('entities.export_pdf') }}.pdf
  • {{ trans('entities.export_text') }}.txt
  • {{ trans('entities.export_md') }}.md
    • +
  • {{ trans('entities.export_zip') }}.zip
    • diff --git a/routes/web.php b/routes/web.php index 5220684c0..6ae70983d 100644 --- a/routes/web.php +++ b/routes/web.php @@ -91,6 +91,7 @@ Route::middleware('auth')->group(function () { Route::get('/books/{bookSlug}/page/{pageSlug}/export/html', [ExportControllers\PageExportController::class, 'html']); Route::get('/books/{bookSlug}/page/{pageSlug}/export/markdown', [ExportControllers\PageExportController::class, 'markdown']); Route::get('/books/{bookSlug}/page/{pageSlug}/export/plaintext', [ExportControllers\PageExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/page/{pageSlug}/export/zip', [ExportControllers\PageExportController::class, 'zip']); Route::get('/books/{bookSlug}/page/{pageSlug}/edit', [EntityControllers\PageController::class, 'edit']); Route::get('/books/{bookSlug}/page/{pageSlug}/move', [EntityControllers\PageController::class, 'showMove']); Route::put('/books/{bookSlug}/page/{pageSlug}/move', [EntityControllers\PageController::class, 'move']); diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php new file mode 100644 index 000000000..d8ce00be3 --- /dev/null +++ b/tests/Exports/ZipExportTest.php @@ -0,0 +1,15 @@ +entities->page(); + // TODO + } +} From 7c39dd5cba7b72184adb96a8236ca1a3c99f03e3 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 20 Oct 2024 19:56:56 +0100 Subject: [PATCH 07/41] ZIP Export: Started building link/ref handling --- app/Exports/ZipExportBuilder.php | 56 +++----------- .../ZipExportModels/ZipExportAttachment.php | 37 +++++++++ .../ZipExportModels/ZipExportImage.php | 11 +++ .../ZipExportModels/ZipExportModel.php | 11 +++ app/Exports/ZipExportModels/ZipExportPage.php | 39 ++++++++++ app/Exports/ZipExportModels/ZipExportTag.php | 27 +++++++ app/Exports/ZipExportReferences.php | 55 ++++++++++++++ app/Exports/ZipReferenceParser.php | 75 +++++++++++++++++++ dev/docs/portable-zip-file-format.md | 1 + 9 files changed, 266 insertions(+), 46 deletions(-) create mode 100644 app/Exports/ZipExportModels/ZipExportAttachment.php create mode 100644 app/Exports/ZipExportModels/ZipExportImage.php create mode 100644 app/Exports/ZipExportModels/ZipExportModel.php create mode 100644 app/Exports/ZipExportModels/ZipExportPage.php create mode 100644 app/Exports/ZipExportModels/ZipExportTag.php create mode 100644 app/Exports/ZipExportReferences.php create mode 100644 app/Exports/ZipReferenceParser.php diff --git a/app/Exports/ZipExportBuilder.php b/app/Exports/ZipExportBuilder.php index 2b8b45d0d..720b4997d 100644 --- a/app/Exports/ZipExportBuilder.php +++ b/app/Exports/ZipExportBuilder.php @@ -2,10 +2,9 @@ namespace BookStack\Exports; -use BookStack\Activity\Models\Tag; use BookStack\Entities\Models\Page; use BookStack\Exceptions\ZipExportException; -use BookStack\Uploads\Attachment; +use BookStack\Exports\ZipExportModels\ZipExportPage; use ZipArchive; class ZipExportBuilder @@ -13,7 +12,8 @@ class ZipExportBuilder protected array $data = []; public function __construct( - protected ZipExportFiles $files + protected ZipExportFiles $files, + protected ZipExportReferences $references, ) { } @@ -22,57 +22,21 @@ class ZipExportBuilder */ public function buildForPage(Page $page): string { - $this->data['page'] = $this->convertPage($page); + $exportPage = ZipExportPage::fromModel($page, $this->files); + $this->data['page'] = $exportPage; + + $this->references->addPage($exportPage); + return $this->build(); } - protected function convertPage(Page $page): array - { - $tags = array_map($this->convertTag(...), $page->tags()->get()->all()); - $attachments = array_map($this->convertAttachment(...), $page->attachments()->get()->all()); - - return [ - 'id' => $page->id, - 'name' => $page->name, - 'html' => '', // TODO - 'markdown' => '', // TODO - 'priority' => $page->priority, - 'attachments' => $attachments, - 'images' => [], // TODO - 'tags' => $tags, - ]; - } - - protected function convertAttachment(Attachment $attachment): array - { - $data = [ - 'name' => $attachment->name, - 'order' => $attachment->order, - ]; - - if ($attachment->external) { - $data['link'] = $attachment->path; - } else { - $data['file'] = $this->files->referenceForAttachment($attachment); - } - - return $data; - } - - protected function convertTag(Tag $tag): array - { - return [ - 'name' => $tag->name, - 'value' => $tag->value, - 'order' => $tag->order, - ]; - } - /** * @throws ZipExportException */ protected function build(): string { + $this->references->buildReferences(); + $this->data['exported_at'] = date(DATE_ATOM); $this->data['instance'] = [ 'version' => trim(file_get_contents(base_path('version'))), diff --git a/app/Exports/ZipExportModels/ZipExportAttachment.php b/app/Exports/ZipExportModels/ZipExportAttachment.php new file mode 100644 index 000000000..d6d674a91 --- /dev/null +++ b/app/Exports/ZipExportModels/ZipExportAttachment.php @@ -0,0 +1,37 @@ +id = $model->id; + $instance->name = $model->name; + + if ($model->external) { + $instance->link = $model->path; + } else { + $instance->file = $files->referenceForAttachment($model); + } + + return $instance; + } + + public static function fromModelArray(array $attachmentArray, ZipExportFiles $files): array + { + return array_values(array_map(function (Attachment $attachment) use ($files) { + return self::fromModel($attachment, $files); + }, $attachmentArray)); + } +} diff --git a/app/Exports/ZipExportModels/ZipExportImage.php b/app/Exports/ZipExportModels/ZipExportImage.php new file mode 100644 index 000000000..73fe3bbf5 --- /dev/null +++ b/app/Exports/ZipExportModels/ZipExportImage.php @@ -0,0 +1,11 @@ +id = $model->id; + $instance->name = $model->name; + $instance->html = (new PageContent($model))->render(); + + if (!empty($model->markdown)) { + $instance->markdown = $model->markdown; + } + + $instance->tags = ZipExportTag::fromModelArray($model->tags()->get()->all()); + $instance->attachments = ZipExportAttachment::fromModelArray($model->attachments()->get()->all(), $files); + + return $instance; + } +} diff --git a/app/Exports/ZipExportModels/ZipExportTag.php b/app/Exports/ZipExportModels/ZipExportTag.php new file mode 100644 index 000000000..636c9ff6d --- /dev/null +++ b/app/Exports/ZipExportModels/ZipExportTag.php @@ -0,0 +1,27 @@ +name = $model->name; + $instance->value = $model->value; + $instance->order = $model->order; + + return $instance; + } + + public static function fromModelArray(array $tagArray): array + { + return array_values(array_map(self::fromModel(...), $tagArray)); + } +} diff --git a/app/Exports/ZipExportReferences.php b/app/Exports/ZipExportReferences.php new file mode 100644 index 000000000..89deb7eda --- /dev/null +++ b/app/Exports/ZipExportReferences.php @@ -0,0 +1,55 @@ +id) { + $this->pages[$page->id] = $page; + } + + foreach ($page->attachments as $attachment) { + if ($attachment->id) { + $this->attachments[$attachment->id] = $attachment; + } + } + } + + public function buildReferences(): void + { + // TODO - References to images, attachments, other entities + + // TODO - Parse page MD & HTML + foreach ($this->pages as $page) { + $page->html = $this->parser->parse($page->html ?? '', function (Model $model): ?string { + // TODO - Handle found link to $model + // - Validate we can see/access $model, or/and that it's + // part of the export in progress. + return '[CAT]'; + }); + // TODO - markdown + } + + // TODO - Parse chapter desc html + // TODO - Parse book desc html + } +} diff --git a/app/Exports/ZipReferenceParser.php b/app/Exports/ZipReferenceParser.php new file mode 100644 index 000000000..6ca826bc3 --- /dev/null +++ b/app/Exports/ZipReferenceParser.php @@ -0,0 +1,75 @@ +modelResolvers = [ + new PagePermalinkModelResolver($queries->pages), + new PageLinkModelResolver($queries->pages), + new ChapterLinkModelResolver($queries->chapters), + new BookLinkModelResolver($queries->books), + // TODO - Image + // TODO - Attachment + ]; + } + + /** + * Parse and replace references in the given content. + * @param callable(Model):(string|null) $handler + */ + public function parse(string $content, callable $handler): string + { + $escapedBase = preg_quote(url('/'), '/'); + $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#]/"; + $matches = []; + preg_match_all($linkRegex, $content, $matches); + + if (count($matches) < 2) { + return $content; + } + + foreach ($matches[1] as $link) { + $model = $this->linkToModel($link); + if ($model) { + $result = $handler($model); + if ($result !== null) { + $content = str_replace($link, $result, $content); + } + } + } + + return $content; + } + + + /** + * Attempt to resolve the given link to a model using the instance model resolvers. + */ + protected function linkToModel(string $link): ?Model + { + foreach ($this->modelResolvers as $resolver) { + $model = $resolver->resolve($link); + if (!is_null($model)) { + return $model; + } + } + + return null; + } +} diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index d5635bd39..7a99563d1 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -128,6 +128,7 @@ File must be an image type accepted by BookStack (png, jpg, gif, webp) #### Attachment +- `id` - Number, optional, original ID for the attachment from exported system. - `name` - String, required, name of attachment. - `link` - String, semi-optional, URL of attachment. - `file` - String reference, semi-optional, reference to attachment file. From 06ffd8ee721a74dea9c584002b2793cc68c873a0 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 21 Oct 2024 12:13:41 +0100 Subject: [PATCH 08/41] Zip Exports: Added attachment/image link resolving & JSON null handling --- .../ZipExportModels/ZipExportAttachment.php | 2 +- .../ZipExportModels/ZipExportImage.php | 2 +- .../ZipExportModels/ZipExportModel.php | 17 +++++++--- app/Exports/ZipExportModels/ZipExportPage.php | 2 +- app/Exports/ZipExportModels/ZipExportTag.php | 2 +- app/Exports/ZipExportReferences.php | 3 ++ app/Exports/ZipReferenceParser.php | 6 ++-- .../AttachmentModelResolver.php | 22 +++++++++++++ .../ModelResolvers/ImageModelResolver.php | 33 +++++++++++++++++++ 9 files changed, 79 insertions(+), 10 deletions(-) create mode 100644 app/References/ModelResolvers/AttachmentModelResolver.php create mode 100644 app/References/ModelResolvers/ImageModelResolver.php diff --git a/app/Exports/ZipExportModels/ZipExportAttachment.php b/app/Exports/ZipExportModels/ZipExportAttachment.php index d6d674a91..d79a16cc1 100644 --- a/app/Exports/ZipExportModels/ZipExportAttachment.php +++ b/app/Exports/ZipExportModels/ZipExportAttachment.php @@ -5,7 +5,7 @@ namespace BookStack\Exports\ZipExportModels; use BookStack\Exports\ZipExportFiles; use BookStack\Uploads\Attachment; -class ZipExportAttachment implements ZipExportModel +class ZipExportAttachment extends ZipExportModel { public ?int $id = null; public string $name; diff --git a/app/Exports/ZipExportModels/ZipExportImage.php b/app/Exports/ZipExportModels/ZipExportImage.php index 73fe3bbf5..540d3d4e5 100644 --- a/app/Exports/ZipExportModels/ZipExportImage.php +++ b/app/Exports/ZipExportModels/ZipExportImage.php @@ -4,7 +4,7 @@ namespace BookStack\Exports\ZipExportModels; use BookStack\Activity\Models\Tag; -class ZipExportImage implements ZipExportModel +class ZipExportImage extends ZipExportModel { public string $name; public string $file; diff --git a/app/Exports/ZipExportModels/ZipExportModel.php b/app/Exports/ZipExportModels/ZipExportModel.php index e1cb616de..26b994c01 100644 --- a/app/Exports/ZipExportModels/ZipExportModel.php +++ b/app/Exports/ZipExportModels/ZipExportModel.php @@ -2,10 +2,19 @@ namespace BookStack\Exports\ZipExportModels; -use BookStack\App\Model; -use BookStack\Exports\ZipExportFiles; +use JsonSerializable; -interface ZipExportModel +abstract class ZipExportModel implements JsonSerializable { -// public static function fromModel(Model $model, ZipExportFiles $files): self; + /** + * Handle the serialization to JSON. + * For these exports, we filter out optional (represented as nullable) fields + * just to clean things up and prevent confusion to avoid null states in the + * resulting export format itself. + */ + public function jsonSerialize(): array + { + $publicProps = get_object_vars(...)->__invoke($this); + return array_filter($publicProps, fn ($value) => $value !== null); + } } diff --git a/app/Exports/ZipExportModels/ZipExportPage.php b/app/Exports/ZipExportModels/ZipExportPage.php index 6589ce60a..c7a950354 100644 --- a/app/Exports/ZipExportModels/ZipExportPage.php +++ b/app/Exports/ZipExportModels/ZipExportPage.php @@ -6,7 +6,7 @@ use BookStack\Entities\Models\Page; use BookStack\Entities\Tools\PageContent; use BookStack\Exports\ZipExportFiles; -class ZipExportPage implements ZipExportModel +class ZipExportPage extends ZipExportModel { public ?int $id = null; public string $name; diff --git a/app/Exports/ZipExportModels/ZipExportTag.php b/app/Exports/ZipExportModels/ZipExportTag.php index 636c9ff6d..09ae9f06c 100644 --- a/app/Exports/ZipExportModels/ZipExportTag.php +++ b/app/Exports/ZipExportModels/ZipExportTag.php @@ -4,7 +4,7 @@ namespace BookStack\Exports\ZipExportModels; use BookStack\Activity\Models\Tag; -class ZipExportTag implements ZipExportModel +class ZipExportTag extends ZipExportModel { public string $name; public ?string $value = null; diff --git a/app/Exports/ZipExportReferences.php b/app/Exports/ZipExportReferences.php index 89deb7eda..76a7fedbe 100644 --- a/app/Exports/ZipExportReferences.php +++ b/app/Exports/ZipExportReferences.php @@ -44,11 +44,14 @@ class ZipExportReferences // TODO - Handle found link to $model // - Validate we can see/access $model, or/and that it's // part of the export in progress. + + // TODO - Add images after the above to files return '[CAT]'; }); // TODO - markdown } +// dd('end'); // TODO - Parse chapter desc html // TODO - Parse book desc html } diff --git a/app/Exports/ZipReferenceParser.php b/app/Exports/ZipReferenceParser.php index 6ca826bc3..820920da2 100644 --- a/app/Exports/ZipReferenceParser.php +++ b/app/Exports/ZipReferenceParser.php @@ -4,9 +4,11 @@ namespace BookStack\Exports; use BookStack\App\Model; use BookStack\Entities\Queries\EntityQueries; +use BookStack\References\ModelResolvers\AttachmentModelResolver; use BookStack\References\ModelResolvers\BookLinkModelResolver; use BookStack\References\ModelResolvers\ChapterLinkModelResolver; use BookStack\References\ModelResolvers\CrossLinkModelResolver; +use BookStack\References\ModelResolvers\ImageModelResolver; use BookStack\References\ModelResolvers\PageLinkModelResolver; use BookStack\References\ModelResolvers\PagePermalinkModelResolver; @@ -24,8 +26,8 @@ class ZipReferenceParser new PageLinkModelResolver($queries->pages), new ChapterLinkModelResolver($queries->chapters), new BookLinkModelResolver($queries->books), - // TODO - Image - // TODO - Attachment + new ImageModelResolver(), + new AttachmentModelResolver(), ]; } diff --git a/app/References/ModelResolvers/AttachmentModelResolver.php b/app/References/ModelResolvers/AttachmentModelResolver.php new file mode 100644 index 000000000..e870d515b --- /dev/null +++ b/app/References/ModelResolvers/AttachmentModelResolver.php @@ -0,0 +1,22 @@ +find($id); + } +} diff --git a/app/References/ModelResolvers/ImageModelResolver.php b/app/References/ModelResolvers/ImageModelResolver.php new file mode 100644 index 000000000..331dd593b --- /dev/null +++ b/app/References/ModelResolvers/ImageModelResolver.php @@ -0,0 +1,33 @@ +where('path', '=', $fullPath)->first(); + } +} From 4fb4fe0931d220ffb9d7e173388351047b665f4c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 21 Oct 2024 13:59:15 +0100 Subject: [PATCH 09/41] ZIP Exports: Added working image handling/inclusion --- app/Exports/ZipExportBuilder.php | 2 +- app/Exports/ZipExportFiles.php | 51 ++++++++++++++- .../ZipExportModels/ZipExportImage.php | 16 ++++- app/Exports/ZipExportReferences.php | 62 ++++++++++++++++--- app/Uploads/ImageService.php | 13 ++++ app/Uploads/ImageStorageDisk.php | 9 +++ dev/docs/portable-zip-file-format.md | 13 ++-- 7 files changed, 148 insertions(+), 18 deletions(-) diff --git a/app/Exports/ZipExportBuilder.php b/app/Exports/ZipExportBuilder.php index 720b4997d..5c56e531b 100644 --- a/app/Exports/ZipExportBuilder.php +++ b/app/Exports/ZipExportBuilder.php @@ -35,7 +35,7 @@ class ZipExportBuilder */ protected function build(): string { - $this->references->buildReferences(); + $this->references->buildReferences($this->files); $this->data['exported_at'] = date(DATE_ATOM); $this->data['instance'] = [ diff --git a/app/Exports/ZipExportFiles.php b/app/Exports/ZipExportFiles.php index d3ee70e93..27b6f937a 100644 --- a/app/Exports/ZipExportFiles.php +++ b/app/Exports/ZipExportFiles.php @@ -4,6 +4,8 @@ namespace BookStack\Exports; use BookStack\Uploads\Attachment; use BookStack\Uploads\AttachmentService; +use BookStack\Uploads\Image; +use BookStack\Uploads\ImageService; use Illuminate\Support\Str; class ZipExportFiles @@ -14,8 +16,15 @@ class ZipExportFiles */ protected array $attachmentRefsById = []; + /** + * References for images by image ID. + * @var array + */ + protected array $imageRefsById = []; + public function __construct( protected AttachmentService $attachmentService, + protected ImageService $imageService, ) { } @@ -30,15 +39,46 @@ class ZipExportFiles return $this->attachmentRefsById[$attachment->id]; } + $existingFiles = $this->getAllFileNames(); do { $fileName = Str::random(20) . '.' . $attachment->extension; - } while (in_array($fileName, $this->attachmentRefsById)); + } while (in_array($fileName, $existingFiles)); $this->attachmentRefsById[$attachment->id] = $fileName; return $fileName; } + /** + * Gain a reference to the given image instance. + * This is expected to be an image that the user has visibility of, + * no permission/access checks are performed here. + */ + public function referenceForImage(Image $image): string + { + if (isset($this->imageRefsById[$image->id])) { + return $this->imageRefsById[$image->id]; + } + + $existingFiles = $this->getAllFileNames(); + $extension = pathinfo($image->path, PATHINFO_EXTENSION); + do { + $fileName = Str::random(20) . '.' . $extension; + } while (in_array($fileName, $existingFiles)); + + $this->imageRefsById[$image->id] = $fileName; + + return $fileName; + } + + protected function getAllFileNames(): array + { + return array_merge( + array_values($this->attachmentRefsById), + array_values($this->imageRefsById), + ); + } + /** * Extract each of the ZIP export tracked files. * Calls the given callback for each tracked file, passing a temporary @@ -54,5 +94,14 @@ class ZipExportFiles stream_copy_to_stream($stream, $tmpFileStream); $callback($tmpFile, $ref); } + + foreach ($this->imageRefsById as $imageId => $ref) { + $image = Image::query()->find($imageId); + $stream = $this->imageService->getImageStream($image); + $tmpFile = tempnam(sys_get_temp_dir(), 'bszipimage-'); + $tmpFileStream = fopen($tmpFile, 'w'); + stream_copy_to_stream($stream, $tmpFileStream); + $callback($tmpFile, $ref); + } } } diff --git a/app/Exports/ZipExportModels/ZipExportImage.php b/app/Exports/ZipExportModels/ZipExportImage.php index 540d3d4e5..39f1d1012 100644 --- a/app/Exports/ZipExportModels/ZipExportImage.php +++ b/app/Exports/ZipExportModels/ZipExportImage.php @@ -2,10 +2,24 @@ namespace BookStack\Exports\ZipExportModels; -use BookStack\Activity\Models\Tag; +use BookStack\Exports\ZipExportFiles; +use BookStack\Uploads\Image; class ZipExportImage extends ZipExportModel { + public ?int $id = null; public string $name; public string $file; + public string $type; + + public static function fromModel(Image $model, ZipExportFiles $files): self + { + $instance = new self(); + $instance->id = $model->id; + $instance->name = $model->name; + $instance->type = $model->type; + $instance->file = $files->referenceForImage($model); + + return $instance; + } } diff --git a/app/Exports/ZipExportReferences.php b/app/Exports/ZipExportReferences.php index 76a7fedbe..19672db0a 100644 --- a/app/Exports/ZipExportReferences.php +++ b/app/Exports/ZipExportReferences.php @@ -3,8 +3,13 @@ namespace BookStack\Exports; use BookStack\App\Model; +use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExportModels\ZipExportAttachment; +use BookStack\Exports\ZipExportModels\ZipExportImage; +use BookStack\Exports\ZipExportModels\ZipExportModel; use BookStack\Exports\ZipExportModels\ZipExportPage; +use BookStack\Uploads\Attachment; +use BookStack\Uploads\Image; class ZipExportReferences { @@ -16,6 +21,9 @@ class ZipExportReferences /** @var ZipExportAttachment[] */ protected array $attachments = []; + /** @var ZipExportImage[] */ + protected array $images = []; + public function __construct( protected ZipReferenceParser $parser, ) { @@ -34,19 +42,12 @@ class ZipExportReferences } } - public function buildReferences(): void + public function buildReferences(ZipExportFiles $files): void { - // TODO - References to images, attachments, other entities - // TODO - Parse page MD & HTML foreach ($this->pages as $page) { - $page->html = $this->parser->parse($page->html ?? '', function (Model $model): ?string { - // TODO - Handle found link to $model - // - Validate we can see/access $model, or/and that it's - // part of the export in progress. - - // TODO - Add images after the above to files - return '[CAT]'; + $page->html = $this->parser->parse($page->html ?? '', function (Model $model) use ($files, $page) { + return $this->handleModelReference($model, $page, $files); }); // TODO - markdown } @@ -55,4 +56,45 @@ class ZipExportReferences // TODO - Parse chapter desc html // TODO - Parse book desc html } + + protected function handleModelReference(Model $model, ZipExportModel $exportModel, ZipExportFiles $files): ?string + { + // TODO - References to other entities + + // Handle attachment references + // No permission check needed here since they would only already exist in this + // reference context if already allowed via their entity access. + if ($model instanceof Attachment) { + if (isset($this->attachments[$model->id])) { + return "[[bsexport:attachment:{$model->id}]]"; + } + return null; + } + + // Handle image references + if ($model instanceof Image) { + // Only handle gallery and drawio images + if ($model->type !== 'gallery' && $model->type !== 'drawio') { + return null; + } + + // We don't expect images to be part of book/chapter content + if (!($exportModel instanceof ZipExportPage)) { + return null; + } + + $page = $model->getPage(); + if ($page && userCan('view', $page)) { + if (!isset($this->images[$model->id])) { + $exportImage = ZipExportImage::fromModel($model, $files); + $this->images[$model->id] = $exportImage; + $exportModel->images[] = $exportImage; + } + return "[[bsexport:image:{$model->id}]]"; + } + return null; + } + + return null; + } } diff --git a/app/Uploads/ImageService.php b/app/Uploads/ImageService.php index 8d8da61ec..e501cc7b1 100644 --- a/app/Uploads/ImageService.php +++ b/app/Uploads/ImageService.php @@ -133,6 +133,19 @@ class ImageService return $disk->get($image->path); } + /** + * Get the raw data content from an image. + * + * @throws Exception + * @returns ?resource + */ + public function getImageStream(Image $image): mixed + { + $disk = $this->storage->getDisk(); + + return $disk->stream($image->path); + } + /** * Destroy an image along with its revisions, thumbnails and remaining folders. * diff --git a/app/Uploads/ImageStorageDisk.php b/app/Uploads/ImageStorageDisk.php index 798b72abd..8df702e0d 100644 --- a/app/Uploads/ImageStorageDisk.php +++ b/app/Uploads/ImageStorageDisk.php @@ -55,6 +55,15 @@ class ImageStorageDisk return $this->filesystem->get($this->adjustPathForDisk($path)); } + /** + * Get a stream to the file at the given path. + * @returns ?resource + */ + public function stream(string $path): mixed + { + return $this->filesystem->readStream($this->adjustPathForDisk($path)); + } + /** * Save the given image data at the given path. Can choose to set * the image as public which will update its visibility after saving. diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 7a99563d1..1ba587201 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -46,13 +46,12 @@ This can be done using the following format: [[bsexport::]] ``` -Images and attachments are referenced via their file name within the `files/` directory. -Otherwise, other content types are referenced by `id`. +References are to the `id` for data objects. Here's an example of each type of such reference that could be used: ``` -[[bsexport:image:an-image-path.png]] -[[bsexport:attachment:an-image-path.png]] +[[bsexport:image:22]] +[[bsexport:attachment:55]] [[bsexport:page:40]] [[bsexport:chapter:2]] [[bsexport:book:8]] @@ -121,10 +120,14 @@ The page editor type, and edit content will be determined by what content is pro #### Image +- `id` - Number, optional, original ID for the page from exported system. - `name` - String, required, name of image. - `file` - String reference, required, reference to image file. +- `type` - String, required, must be 'gallery' or 'drawio' -File must be an image type accepted by BookStack (png, jpg, gif, webp) +File must be an image type accepted by BookStack (png, jpg, gif, webp). +Images of type 'drawio' are expected to be png with draw.io drawing data +embedded within it. #### Attachment From f732ef05d5b60a48ce3b42e05155e9e91d92d927 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 23 Oct 2024 10:48:26 +0100 Subject: [PATCH 10/41] ZIP Exports: Reorganised files, added page md parsing --- .../Controllers/PageExportController.php | 2 +- .../Models}/ZipExportAttachment.php | 4 ++-- .../Models}/ZipExportImage.php | 4 ++-- .../Models}/ZipExportModel.php | 2 +- .../Models}/ZipExportPage.php | 4 ++-- .../Models}/ZipExportTag.php | 2 +- .../{ => ZipExports}/ZipExportBuilder.php | 4 ++-- .../{ => ZipExports}/ZipExportFiles.php | 2 +- .../{ => ZipExports}/ZipExportReferences.php | 23 +++++++++++-------- .../{ => ZipExports}/ZipReferenceParser.php | 2 +- 10 files changed, 26 insertions(+), 23 deletions(-) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportAttachment.php (90%) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportImage.php (84%) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportModel.php (92%) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportPage.php (91%) rename app/Exports/{ZipExportModels => ZipExports/Models}/ZipExportTag.php (92%) rename app/Exports/{ => ZipExports}/ZipExportBuilder.php (94%) rename app/Exports/{ => ZipExports}/ZipExportFiles.php (98%) rename app/Exports/{ => ZipExports}/ZipExportReferences.php (83%) rename app/Exports/{ => ZipExports}/ZipReferenceParser.php (98%) diff --git a/app/Exports/Controllers/PageExportController.php b/app/Exports/Controllers/PageExportController.php index 01611fd21..34e67ffcf 100644 --- a/app/Exports/Controllers/PageExportController.php +++ b/app/Exports/Controllers/PageExportController.php @@ -6,7 +6,7 @@ use BookStack\Entities\Queries\PageQueries; use BookStack\Entities\Tools\PageContent; use BookStack\Exceptions\NotFoundException; use BookStack\Exports\ExportFormatter; -use BookStack\Exports\ZipExportBuilder; +use BookStack\Exports\ZipExports\ZipExportBuilder; use BookStack\Http\Controller; use Throwable; diff --git a/app/Exports/ZipExportModels/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php similarity index 90% rename from app/Exports/ZipExportModels/ZipExportAttachment.php rename to app/Exports/ZipExports/Models/ZipExportAttachment.php index d79a16cc1..8c89ae11f 100644 --- a/app/Exports/ZipExportModels/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -1,8 +1,8 @@ pages as $page) { - $page->html = $this->parser->parse($page->html ?? '', function (Model $model) use ($files, $page) { + $handler = function (Model $model) use ($files, $page) { return $this->handleModelReference($model, $page, $files); - }); - // TODO - markdown + }; + + $page->html = $this->parser->parse($page->html ?? '', $handler); + if ($page->markdown) { + $page->markdown = $this->parser->parse($page->markdown, $handler); + } } // dd('end'); diff --git a/app/Exports/ZipReferenceParser.php b/app/Exports/ZipExports/ZipReferenceParser.php similarity index 98% rename from app/Exports/ZipReferenceParser.php rename to app/Exports/ZipExports/ZipReferenceParser.php index 820920da2..4d16dbc61 100644 --- a/app/Exports/ZipReferenceParser.php +++ b/app/Exports/ZipExports/ZipReferenceParser.php @@ -1,6 +1,6 @@ Date: Wed, 23 Oct 2024 11:30:32 +0100 Subject: [PATCH 11/41] ZIP Exports: Added core logic for books/chapters --- app/Entities/Models/Chapter.php | 1 + .../ZipExports/Models/ZipExportBook.php | 53 ++++++++++++++++ .../ZipExports/Models/ZipExportChapter.php | 45 ++++++++++++++ .../ZipExports/Models/ZipExportPage.php | 12 ++++ app/Exports/ZipExports/ZipExportBuilder.php | 30 +++++++++ .../ZipExports/ZipExportReferences.php | 61 ++++++++++++++++--- dev/docs/portable-zip-file-format.md | 2 +- 7 files changed, 195 insertions(+), 9 deletions(-) create mode 100644 app/Exports/ZipExports/Models/ZipExportBook.php create mode 100644 app/Exports/ZipExports/Models/ZipExportChapter.php diff --git a/app/Entities/Models/Chapter.php b/app/Entities/Models/Chapter.php index c926aaa64..088d199da 100644 --- a/app/Entities/Models/Chapter.php +++ b/app/Entities/Models/Chapter.php @@ -60,6 +60,7 @@ class Chapter extends BookChild /** * Get the visible pages in this chapter. + * @returns Collection */ public function getVisiblePages(): Collection { diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php new file mode 100644 index 000000000..5a0c5806b --- /dev/null +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -0,0 +1,53 @@ +id = $model->id; + $instance->name = $model->name; + $instance->description_html = $model->descriptionHtml(); + + if ($model->cover) { + $instance->cover = $files->referenceForImage($model->cover); + } + + $instance->tags = ZipExportTag::fromModelArray($model->tags()->get()->all()); + + $chapters = []; + $pages = []; + + $children = $model->getDirectVisibleChildren()->all(); + foreach ($children as $child) { + if ($child instanceof Chapter) { + $chapters[] = $child; + } else if ($child instanceof Page) { + $pages[] = $child; + } + } + + $instance->pages = ZipExportPage::fromModelArray($pages, $files); + $instance->chapters = ZipExportChapter::fromModelArray($chapters, $files); + + return $instance; + } +} diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php new file mode 100644 index 000000000..cd5765f48 --- /dev/null +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -0,0 +1,45 @@ +id = $model->id; + $instance->name = $model->name; + $instance->description_html = $model->descriptionHtml(); + $instance->priority = $model->priority; + $instance->tags = ZipExportTag::fromModelArray($model->tags()->get()->all()); + + $pages = $model->getVisiblePages()->filter(fn (Page $page) => !$page->draft)->all(); + $instance->pages = ZipExportPage::fromModelArray($pages, $files); + + return $instance; + } + + /** + * @param Chapter[] $chapterArray + * @return self[] + */ + public static function fromModelArray(array $chapterArray, ZipExportFiles $files): array + { + return array_values(array_map(function (Chapter $chapter) use ($files) { + return self::fromModel($chapter, $files); + }, $chapterArray)); + } +} diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index bae46ca82..8075595f2 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -26,6 +26,7 @@ class ZipExportPage extends ZipExportModel $instance->id = $model->id; $instance->name = $model->name; $instance->html = (new PageContent($model))->render(); + $instance->priority = $model->priority; if (!empty($model->markdown)) { $instance->markdown = $model->markdown; @@ -36,4 +37,15 @@ class ZipExportPage extends ZipExportModel return $instance; } + + /** + * @param Page[] $pageArray + * @return self[] + */ + public static function fromModelArray(array $pageArray, ZipExportFiles $files): array + { + return array_values(array_map(function (Page $page) use ($files) { + return self::fromModel($page, $files); + }, $pageArray)); + } } diff --git a/app/Exports/ZipExports/ZipExportBuilder.php b/app/Exports/ZipExports/ZipExportBuilder.php index 15edebea5..42fb03541 100644 --- a/app/Exports/ZipExports/ZipExportBuilder.php +++ b/app/Exports/ZipExports/ZipExportBuilder.php @@ -2,8 +2,12 @@ namespace BookStack\Exports\ZipExports; +use BookStack\Entities\Models\Book; +use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; use BookStack\Exceptions\ZipExportException; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportPage; use ZipArchive; @@ -30,6 +34,32 @@ class ZipExportBuilder return $this->build(); } + /** + * @throws ZipExportException + */ + public function buildForChapter(Chapter $chapter): string + { + $exportChapter = ZipExportChapter::fromModel($chapter, $this->files); + $this->data['chapter'] = $exportChapter; + + $this->references->addChapter($exportChapter); + + return $this->build(); + } + + /** + * @throws ZipExportException + */ + public function buildForBook(Book $book): string + { + $exportBook = ZipExportBook::fromModel($book, $this->files); + $this->data['book'] = $exportBook; + + $this->references->addBook($exportBook); + + return $this->build(); + } + /** * @throws ZipExportException */ diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index c3565aaa3..1fce0fc97 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -4,6 +4,8 @@ namespace BookStack\Exports\ZipExports; use BookStack\App\Model; use BookStack\Exports\ZipExports\Models\ZipExportAttachment; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportImage; use BookStack\Exports\ZipExports\Models\ZipExportModel; use BookStack\Exports\ZipExports\Models\ZipExportPage; @@ -14,8 +16,10 @@ class ZipExportReferences { /** @var ZipExportPage[] */ protected array $pages = []; - protected array $books = []; + /** @var ZipExportChapter[] */ protected array $chapters = []; + /** @var ZipExportBook[] */ + protected array $books = []; /** @var ZipExportAttachment[] */ protected array $attachments = []; @@ -41,23 +45,64 @@ class ZipExportReferences } } + public function addChapter(ZipExportChapter $chapter): void + { + if ($chapter->id) { + $this->chapters[$chapter->id] = $chapter; + } + + foreach ($chapter->pages as $page) { + $this->addPage($page); + } + } + + public function addBook(ZipExportBook $book): void + { + if ($book->id) { + $this->chapters[$book->id] = $book; + } + + foreach ($book->pages as $page) { + $this->addPage($page); + } + + foreach ($book->chapters as $chapter) { + $this->addChapter($chapter); + } + } + public function buildReferences(ZipExportFiles $files): void { + $createHandler = function (ZipExportModel $zipModel) use ($files) { + return function (Model $model) use ($files, $zipModel) { + return $this->handleModelReference($model, $zipModel, $files); + }; + }; + // Parse page content first foreach ($this->pages as $page) { - $handler = function (Model $model) use ($files, $page) { - return $this->handleModelReference($model, $page, $files); - }; - + $handler = $createHandler($page); $page->html = $this->parser->parse($page->html ?? '', $handler); if ($page->markdown) { $page->markdown = $this->parser->parse($page->markdown, $handler); } } -// dd('end'); - // TODO - Parse chapter desc html - // TODO - Parse book desc html + // Parse chapter description HTML + foreach ($this->chapters as $chapter) { + if ($chapter->description_html) { + $handler = $createHandler($chapter); + $chapter->description_html = $this->parser->parse($chapter->description_html, $handler); + } + } + + // Parse book description HTML + foreach ($this->books as $book) { + if ($book->description_html) { + $handler = $createHandler($book); + $book->description_html = $this->parser->parse($book->description_html, $handler); + } + } } protected function handleModelReference(Model $model, ZipExportModel $exportModel, ZipExportFiles $files): ?string diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 1ba587201..6cee7356d 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -87,7 +87,7 @@ The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This i - `id` - Number, optional, original ID for the book from exported system. - `name` - String, required, name/title of the book. - `description_html` - String, optional, HTML description content. -- `cover` - String reference, options, reference to book cover image. +- `cover` - String reference, optional, reference to book cover image. - `chapters` - [Chapter](#chapter) array, optional, chapters within this book. - `pages` - [Page](#page) array, optional, direct child pages for this book. - `tags` - [Tag](#tag) array, optional, tags assigned to this book. From 484342f26adab723b8c4625d22a8901f5bfe79af Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 23 Oct 2024 15:59:58 +0100 Subject: [PATCH 12/41] ZIP Exports: Added entity cross refs, Started export tests --- .../Controllers/BookExportController.php | 14 +++ .../Controllers/ChapterExportController.php | 13 +++ .../ZipExports/ZipExportReferences.php | 14 ++- routes/web.php | 1 + tests/Exports/ZipExportTest.php | 85 ++++++++++++++++++- tests/Exports/ZipResultData.php | 13 +++ 6 files changed, 136 insertions(+), 4 deletions(-) create mode 100644 tests/Exports/ZipResultData.php diff --git a/app/Exports/Controllers/BookExportController.php b/app/Exports/Controllers/BookExportController.php index 36906b6ad..f726175a0 100644 --- a/app/Exports/Controllers/BookExportController.php +++ b/app/Exports/Controllers/BookExportController.php @@ -3,7 +3,9 @@ namespace BookStack\Exports\Controllers; use BookStack\Entities\Queries\BookQueries; +use BookStack\Exceptions\NotFoundException; use BookStack\Exports\ExportFormatter; +use BookStack\Exports\ZipExports\ZipExportBuilder; use BookStack\Http\Controller; use Throwable; @@ -63,4 +65,16 @@ class BookExportController extends Controller return $this->download()->directly($textContent, $bookSlug . '.md'); } + + /** + * Export a book to a contained ZIP export file. + * @throws NotFoundException + */ + public function zip(string $bookSlug, ZipExportBuilder $builder) + { + $book = $this->queries->findVisibleBySlugOrFail($bookSlug); + $zip = $builder->buildForBook($book); + + return $this->download()->streamedDirectly(fopen($zip, 'r'), $bookSlug . '.zip', filesize($zip)); + } } diff --git a/app/Exports/Controllers/ChapterExportController.php b/app/Exports/Controllers/ChapterExportController.php index d85b90dcb..0d7a5c0d1 100644 --- a/app/Exports/Controllers/ChapterExportController.php +++ b/app/Exports/Controllers/ChapterExportController.php @@ -5,6 +5,7 @@ namespace BookStack\Exports\Controllers; use BookStack\Entities\Queries\ChapterQueries; use BookStack\Exceptions\NotFoundException; use BookStack\Exports\ExportFormatter; +use BookStack\Exports\ZipExports\ZipExportBuilder; use BookStack\Http\Controller; use Throwable; @@ -70,4 +71,16 @@ class ChapterExportController extends Controller return $this->download()->directly($chapterText, $chapterSlug . '.md'); } + + /** + * Export a book to a contained ZIP export file. + * @throws NotFoundException + */ + public function zip(string $bookSlug, string $chapterSlug, ZipExportBuilder $builder) + { + $chapter = $this->queries->findVisibleBySlugsOrFail($bookSlug, $chapterSlug); + $zip = $builder->buildForChapter($chapter); + + return $this->download()->streamedDirectly(fopen($zip, 'r'), $chapterSlug . '.zip', filesize($zip)); + } } diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index 1fce0fc97..8b3a4b612 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -3,6 +3,9 @@ namespace BookStack\Exports\ZipExports; use BookStack\App\Model; +use BookStack\Entities\Models\Book; +use BookStack\Entities\Models\Chapter; +use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\Models\ZipExportAttachment; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; @@ -107,8 +110,6 @@ class ZipExportReferences protected function handleModelReference(Model $model, ZipExportModel $exportModel, ZipExportFiles $files): ?string { - // TODO - References to other entities - // Handle attachment references // No permission check needed here since they would only already exist in this // reference context if already allowed via their entity access. @@ -143,6 +144,15 @@ class ZipExportReferences return null; } + // Handle entity references + if ($model instanceof Book && isset($this->books[$model->id])) { + return "[[bsexport:book:{$model->id}]]"; + } else if ($model instanceof Chapter && isset($this->chapters[$model->id])) { + return "[[bsexport:chapter:{$model->id}]]"; + } else if ($model instanceof Page && isset($this->pages[$model->id])) { + return "[[bsexport:page:{$model->id}]]"; + } + return null; } } diff --git a/routes/web.php b/routes/web.php index 6ae70983d..e6f3683c6 100644 --- a/routes/web.php +++ b/routes/web.php @@ -132,6 +132,7 @@ Route::middleware('auth')->group(function () { Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/html', [ExportControllers\ChapterExportController::class, 'html']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/markdown', [ExportControllers\ChapterExportController::class, 'markdown']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/plaintext', [ExportControllers\ChapterExportController::class, 'plainText']); + Route::get('/books/{bookSlug}/chapter/{chapterSlug}/export/zip', [ExportControllers\ChapterExportController::class, 'zip']); Route::put('/books/{bookSlug}/chapter/{chapterSlug}/permissions', [PermissionsController::class, 'updateForChapter']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/references', [ReferenceController::class, 'chapter']); Route::get('/books/{bookSlug}/chapter/{chapterSlug}/delete', [EntityControllers\ChapterController::class, 'showDelete']); diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index d8ce00be3..536e23806 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -2,14 +2,95 @@ namespace Tests\Exports; -use BookStack\Entities\Models\Book; +use Illuminate\Support\Carbon; +use Illuminate\Testing\TestResponse; use Tests\TestCase; +use ZipArchive; class ZipExportTest extends TestCase { - public function test_page_export() + public function test_export_results_in_zip_format() { $page = $this->entities->page(); + $response = $this->asEditor()->get($page->getUrl("/export/zip")); + + $zipData = $response->streamedContent(); + $zipFile = tempnam(sys_get_temp_dir(), 'bstesta-'); + file_put_contents($zipFile, $zipData); + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::RDONLY); + + $this->assertNotFalse($zip->locateName('data.json')); + $this->assertNotFalse($zip->locateName('files/')); + + $data = json_decode($zip->getFromName('data.json'), true); + $this->assertIsArray($data); + $this->assertGreaterThan(0, count($data)); + + $zip->close(); + unlink($zipFile); + } + + public function test_export_metadata() + { + $page = $this->entities->page(); + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $this->assertEquals($page->id, $zip->data['page']['id'] ?? null); + $this->assertArrayNotHasKey('book', $zip->data); + $this->assertArrayNotHasKey('chapter', $zip->data); + + $now = time(); + $date = Carbon::parse($zip->data['exported_at'])->unix(); + $this->assertLessThan($now + 2, $date); + $this->assertGreaterThan($now - 2, $date); + + $version = trim(file_get_contents(base_path('version'))); + $this->assertEquals($version, $zip->data['instance']['version']); + + $instanceId = decrypt($zip->data['instance']['id_ciphertext']); + $this->assertEquals('bookstack', $instanceId); + } + + public function test_page_export() + { // TODO } + + public function test_book_export() + { + // TODO + } + + public function test_chapter_export() + { + // TODO + } + + protected function extractZipResponse(TestResponse $response): ZipResultData + { + $zipData = $response->streamedContent(); + $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); + + file_put_contents($zipFile, $zipData); + $extractDir = tempnam(sys_get_temp_dir(), 'bstestextracted-'); + if (file_exists($extractDir)) { + unlink($extractDir); + } + mkdir($extractDir); + + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::RDONLY); + $zip->extractTo($extractDir); + + $dataJson = file_get_contents($extractDir . DIRECTORY_SEPARATOR . "data.json"); + $data = json_decode($dataJson, true); + + return new ZipResultData( + $zipFile, + $extractDir, + $data, + ); + } } diff --git a/tests/Exports/ZipResultData.php b/tests/Exports/ZipResultData.php new file mode 100644 index 000000000..b5cc2b4ca --- /dev/null +++ b/tests/Exports/ZipResultData.php @@ -0,0 +1,13 @@ + Date: Sun, 27 Oct 2024 14:33:43 +0000 Subject: [PATCH 13/41] ZIP Exports: Tested each type and model of export --- .../ZipExports/Models/ZipExportAttachment.php | 1 + .../ZipExports/ZipExportReferences.php | 2 +- app/Exports/ZipExports/ZipReferenceParser.php | 2 +- tests/Exports/ZipExportTest.php | 265 +++++++++++++++++- tests/Exports/ZipResultData.php | 9 + 5 files changed, 274 insertions(+), 5 deletions(-) diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index 8c89ae11f..283ffa751 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -18,6 +18,7 @@ class ZipExportAttachment extends ZipExportModel $instance = new self(); $instance->id = $model->id; $instance->name = $model->name; + $instance->order = $model->order; if ($model->external) { $instance->link = $model->path; diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index 8b3a4b612..c630c832b 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -62,7 +62,7 @@ class ZipExportReferences public function addBook(ZipExportBook $book): void { if ($book->id) { - $this->chapters[$book->id] = $book; + $this->books[$book->id] = $book; } foreach ($book->pages as $page) { diff --git a/app/Exports/ZipExports/ZipReferenceParser.php b/app/Exports/ZipExports/ZipReferenceParser.php index 4d16dbc61..da43d1b36 100644 --- a/app/Exports/ZipExports/ZipReferenceParser.php +++ b/app/Exports/ZipExports/ZipReferenceParser.php @@ -38,7 +38,7 @@ class ZipReferenceParser public function parse(string $content, callable $handler): string { $escapedBase = preg_quote(url('/'), '/'); - $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#]/"; + $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#()]/"; $matches = []; preg_match_all($linkRegex, $content, $matches); diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index 536e23806..ac07b33ae 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -2,6 +2,11 @@ namespace Tests\Exports; +use BookStack\Activity\Models\Tag; +use BookStack\Entities\Repos\BookRepo; +use BookStack\Entities\Tools\PageContent; +use BookStack\Uploads\Attachment; +use BookStack\Uploads\Image; use Illuminate\Support\Carbon; use Illuminate\Testing\TestResponse; use Tests\TestCase; @@ -55,17 +60,271 @@ class ZipExportTest extends TestCase public function test_page_export() { - // TODO + $page = $this->entities->page(); + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertEquals([ + 'id' => $page->id, + 'name' => $page->name, + 'html' => (new PageContent($page))->render(), + 'priority' => $page->priority, + 'attachments' => [], + 'images' => [], + 'tags' => [], + ], $pageData); + } + + public function test_page_export_with_markdown() + { + $page = $this->entities->page(); + $markdown = "# My page\n\nwritten in markdown for export\n"; + $page->markdown = $markdown; + $page->save(); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertEquals($markdown, $pageData['markdown']); + $this->assertNotEmpty($pageData['html']); + } + + public function test_page_export_with_tags() + { + $page = $this->entities->page(); + $page->tags()->saveMany([ + new Tag(['name' => 'Exporty', 'value' => 'Content', 'order' => 1]), + new Tag(['name' => 'Another', 'value' => '', 'order' => 2]), + ]); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertEquals([ + [ + 'name' => 'Exporty', + 'value' => 'Content', + 'order' => 1, + ], + [ + 'name' => 'Another', + 'value' => '', + 'order' => 2, + ] + ], $pageData['tags']); + } + + public function test_page_export_with_images() + { + $this->asEditor(); + $page = $this->entities->page(); + $result = $this->files->uploadGalleryImageToPage($this, $page); + $displayThumb = $result['response']->thumbs->gallery ?? ''; + $page->html = '

    My image

    '; + $page->save(); + $image = Image::findOrFail($result['response']->id); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['page']; + + $this->assertCount(1, $pageData['images']); + $imageData = $pageData['images'][0]; + $this->assertEquals($image->id, $imageData['id']); + $this->assertEquals($image->name, $imageData['name']); + $this->assertEquals('gallery', $imageData['type']); + $this->assertNotEmpty($imageData['file']); + + $filePath = $zip->extractPath("files/{$imageData['file']}"); + $this->assertFileExists($filePath); + $this->assertEquals(file_get_contents(public_path($image->path)), file_get_contents($filePath)); + + $this->assertEquals('

    My image

    ', $pageData['html']); + } + + public function test_page_export_file_attachments() + { + $contents = 'My great attachment content!'; + + $page = $this->entities->page(); + $this->asAdmin(); + $attachment = $this->files->uploadAttachmentDataToPage($this, $page, 'PageAttachmentExport.txt', $contents, 'text/plain'); + + $zipResp = $this->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertCount(1, $pageData['attachments']); + + $attachmentData = $pageData['attachments'][0]; + $this->assertEquals('PageAttachmentExport.txt', $attachmentData['name']); + $this->assertEquals($attachment->id, $attachmentData['id']); + $this->assertEquals(1, $attachmentData['order']); + $this->assertArrayNotHasKey('link', $attachmentData); + $this->assertNotEmpty($attachmentData['file']); + + $fileRef = $attachmentData['file']; + $filePath = $zip->extractPath("/files/$fileRef"); + $this->assertFileExists($filePath); + $this->assertEquals($contents, file_get_contents($filePath)); + } + + public function test_page_export_link_attachments() + { + $page = $this->entities->page(); + $this->asEditor(); + $attachment = Attachment::factory()->create([ + 'name' => 'My link attachment for export', + 'path' => 'https://example.com/cats', + 'external' => true, + 'uploaded_to' => $page->id, + 'order' => 1, + ]); + + $zipResp = $this->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $pageData = $zip->data['page']; + $this->assertCount(1, $pageData['attachments']); + + $attachmentData = $pageData['attachments'][0]; + $this->assertEquals('My link attachment for export', $attachmentData['name']); + $this->assertEquals($attachment->id, $attachmentData['id']); + $this->assertEquals(1, $attachmentData['order']); + $this->assertEquals('https://example.com/cats', $attachmentData['link']); + $this->assertArrayNotHasKey('file', $attachmentData); } public function test_book_export() { - // TODO + $book = $this->entities->book(); + $book->tags()->saveMany(Tag::factory()->count(2)->make()); + + $zipResp = $this->asEditor()->get($book->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $this->assertArrayHasKey('book', $zip->data); + + $bookData = $zip->data['book']; + $this->assertEquals($book->id, $bookData['id']); + $this->assertEquals($book->name, $bookData['name']); + $this->assertEquals($book->descriptionHtml(), $bookData['description_html']); + $this->assertCount(2, $bookData['tags']); + $this->assertCount($book->directPages()->count(), $bookData['pages']); + $this->assertCount($book->chapters()->count(), $bookData['chapters']); + $this->assertArrayNotHasKey('cover', $bookData); + } + + public function test_book_export_with_cover_image() + { + $book = $this->entities->book(); + $bookRepo = $this->app->make(BookRepo::class); + $coverImageFile = $this->files->uploadedImage('cover.png'); + $bookRepo->updateCoverImage($book, $coverImageFile); + $coverImage = $book->cover()->first(); + + $zipResp = $this->asEditor()->get($book->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + + $this->assertArrayHasKey('cover', $zip->data['book']); + $coverRef = $zip->data['book']['cover']; + $coverPath = $zip->extractPath("/files/$coverRef"); + $this->assertFileExists($coverPath); + $this->assertEquals(file_get_contents(public_path($coverImage->path)), file_get_contents($coverPath)); } public function test_chapter_export() { - // TODO + $chapter = $this->entities->chapter(); + $chapter->tags()->saveMany(Tag::factory()->count(2)->make()); + + $zipResp = $this->asEditor()->get($chapter->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $this->assertArrayHasKey('chapter', $zip->data); + + $chapterData = $zip->data['chapter']; + $this->assertEquals($chapter->id, $chapterData['id']); + $this->assertEquals($chapter->name, $chapterData['name']); + $this->assertEquals($chapter->descriptionHtml(), $chapterData['description_html']); + $this->assertCount(2, $chapterData['tags']); + $this->assertEquals($chapter->priority, $chapterData['priority']); + $this->assertCount($chapter->pages()->count(), $chapterData['pages']); + } + + + public function test_cross_reference_links_are_converted() + { + $book = $this->entities->bookHasChaptersAndPages(); + $chapter = $book->chapters()->first(); + $page = $chapter->pages()->first(); + + $book->description_html = '

    Link to chapter

    '; + $book->save(); + $chapter->description_html = '

    Link to page

    '; + $chapter->save(); + $page->html = '

    Link to book

    '; + $page->save(); + + $zipResp = $this->asEditor()->get($book->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $bookData = $zip->data['book']; + $chapterData = $bookData['chapters'][0]; + $pageData = $chapterData['pages'][0]; + + $this->assertStringContainsString('href="[[bsexport:chapter:' . $chapter->id . ']]"', $bookData['description_html']); + $this->assertStringContainsString('href="[[bsexport:page:' . $page->id . ']]#section2"', $chapterData['description_html']); + $this->assertStringContainsString('href="[[bsexport:book:' . $book->id . ']]?view=true"', $pageData['html']); + } + + public function test_cross_reference_links_external_to_export_are_not_converted() + { + $page = $this->entities->page(); + $page->html = '

    Link to book

    '; + $page->save(); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['page']; + + $this->assertStringContainsString('href="' . $page->book->getUrl() . '"', $pageData['html']); + } + + public function test_attachments_links_are_converted() + { + $page = $this->entities->page(); + $attachment = Attachment::factory()->create([ + 'name' => 'My link attachment for export reference', + 'path' => 'https://example.com/cats/ref', + 'external' => true, + 'uploaded_to' => $page->id, + 'order' => 1, + ]); + + $page->html = '

    id}") . '?open=true">Link to attachment

    '; + $page->save(); + + $zipResp = $this->asEditor()->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['page']; + + $this->assertStringContainsString('href="[[bsexport:attachment:' . $attachment->id . ']]?open=true"', $pageData['html']); + } + + public function test_links_in_markdown_are_parsed() + { + $chapter = $this->entities->chapterHasPages(); + $page = $chapter->pages()->first(); + + $page->markdown = "[Link to chapter]({$chapter->getUrl()})"; + $page->save(); + + $zipResp = $this->asEditor()->get($chapter->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['chapter']['pages'][0]; + + $this->assertStringContainsString("[Link to chapter]([[bsexport:chapter:{$chapter->id}]])", $pageData['markdown']); } protected function extractZipResponse(TestResponse $response): ZipResultData diff --git a/tests/Exports/ZipResultData.php b/tests/Exports/ZipResultData.php index b5cc2b4ca..7725004c7 100644 --- a/tests/Exports/ZipResultData.php +++ b/tests/Exports/ZipResultData.php @@ -10,4 +10,13 @@ class ZipResultData public array $data, ) { } + + /** + * Build a path to a location the extracted content, using the given relative $path. + */ + public function extractPath(string $path): string + { + $relPath = implode(DIRECTORY_SEPARATOR, explode('/', $path)); + return $this->extractedDirPath . DIRECTORY_SEPARATOR . ltrim($relPath, DIRECTORY_SEPARATOR); + } } From 4051d5b8037119b382c576042bc668b8f00eee14 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 29 Oct 2024 12:11:51 +0000 Subject: [PATCH 14/41] ZIP Exports: Added new import permission Also updated new route/view to new non-book-specific flow. Also fixed down migration of old export permissions migration. --- app/Exports/Controllers/ImportController.php | 24 ++++++++ ...8_28_161743_add_export_role_permission.php | 7 ++- ...0_29_114420_add_import_role_permission.php | 61 +++++++++++++++++++ lang/en/entities.php | 1 + lang/en/settings.php | 1 + resources/views/books/index.blade.php | 7 +++ resources/views/exports/import.blade.php | 34 +++++++++++ .../views/settings/roles/parts/form.blade.php | 1 + routes/web.php | 4 ++ 9 files changed, 137 insertions(+), 3 deletions(-) create mode 100644 app/Exports/Controllers/ImportController.php create mode 100644 database/migrations/2024_10_29_114420_add_import_role_permission.php create mode 100644 resources/views/exports/import.blade.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php new file mode 100644 index 000000000..acc803a0f --- /dev/null +++ b/app/Exports/Controllers/ImportController.php @@ -0,0 +1,24 @@ +middleware('can:content-import'); + } + + public function start(Request $request) + { + return view('exports.import'); + } + + public function upload(Request $request) + { + // TODO + } +} diff --git a/database/migrations/2021_08_28_161743_add_export_role_permission.php b/database/migrations/2021_08_28_161743_add_export_role_permission.php index 21f45aa06..99416f9fc 100644 --- a/database/migrations/2021_08_28_161743_add_export_role_permission.php +++ b/database/migrations/2021_08_28_161743_add_export_role_permission.php @@ -11,8 +11,7 @@ return new class extends Migration */ public function up(): void { - // Create new templates-manage permission and assign to admin role - $roles = DB::table('roles')->get('id'); + // Create new content-export permission $permissionId = DB::table('role_permissions')->insertGetId([ 'name' => 'content-export', 'display_name' => 'Export Content', @@ -20,6 +19,7 @@ return new class extends Migration 'updated_at' => Carbon::now()->toDateTimeString(), ]); + $roles = DB::table('roles')->get('id'); $permissionRoles = $roles->map(function ($role) use ($permissionId) { return [ 'role_id' => $role->id, @@ -27,6 +27,7 @@ return new class extends Migration ]; })->values()->toArray(); + // Assign to all existing roles in the system DB::table('permission_role')->insert($permissionRoles); } @@ -40,6 +41,6 @@ return new class extends Migration ->where('name', '=', 'content-export')->first(); DB::table('permission_role')->where('permission_id', '=', $contentExportPermission->id)->delete(); - DB::table('role_permissions')->where('id', '=', 'content-export')->delete(); + DB::table('role_permissions')->where('id', '=', $contentExportPermission->id)->delete(); } }; diff --git a/database/migrations/2024_10_29_114420_add_import_role_permission.php b/database/migrations/2024_10_29_114420_add_import_role_permission.php new file mode 100644 index 000000000..17bbe4cff --- /dev/null +++ b/database/migrations/2024_10_29_114420_add_import_role_permission.php @@ -0,0 +1,61 @@ +insertGetId([ + 'name' => 'content-import', + 'display_name' => 'Import Content', + 'created_at' => Carbon::now()->toDateTimeString(), + 'updated_at' => Carbon::now()->toDateTimeString(), + ]); + + // Get existing admin-level role ids + $settingManagePermission = DB::table('role_permissions') + ->where('name', '=', 'settings-manage')->first(); + + if (!$settingManagePermission) { + return; + } + + $adminRoleIds = DB::table('permission_role') + ->where('permission_id', '=', $settingManagePermission->id) + ->pluck('role_id')->all(); + + // Assign the new permission to all existing admins + $newPermissionRoles = array_values(array_map(function ($roleId) use ($permissionId) { + return [ + 'role_id' => $roleId, + 'permission_id' => $permissionId, + ]; + }, $adminRoleIds)); + + DB::table('permission_role')->insert($newPermissionRoles); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + // Remove content-import permission + $importPermission = DB::table('role_permissions') + ->where('name', '=', 'content-import')->first(); + + if (!$importPermission) { + return; + } + + DB::table('permission_role')->where('permission_id', '=', $importPermission->id)->delete(); + DB::table('role_permissions')->where('id', '=', $importPermission->id)->delete(); + } +}; diff --git a/lang/en/entities.php b/lang/en/entities.php index 7e5a708ef..1a61b629a 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -43,6 +43,7 @@ return [ 'default_template' => 'Default Page Template', 'default_template_explain' => 'Assign a page template that will be used as the default content for all pages created within this item. Keep in mind this will only be used if the page creator has view access to the chosen template page.', 'default_template_select' => 'Select a template page', + 'import' => 'Import', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/lang/en/settings.php b/lang/en/settings.php index 5427cb941..c0b6b692a 100644 --- a/lang/en/settings.php +++ b/lang/en/settings.php @@ -162,6 +162,7 @@ return [ 'role_access_api' => 'Access system API', 'role_manage_settings' => 'Manage app settings', 'role_export_content' => 'Export content', + 'role_import_content' => 'Import content', 'role_editor_change' => 'Change page editor', 'role_notifications' => 'Receive & manage notifications', 'role_asset' => 'Asset Permissions', diff --git a/resources/views/books/index.blade.php b/resources/views/books/index.blade.php index 0b407a860..418c0fea8 100644 --- a/resources/views/books/index.blade.php +++ b/resources/views/books/index.blade.php @@ -49,6 +49,13 @@ @icon('tag') {{ trans('entities.tags_view_tags') }} + + @if(userCan('content-import')) + + @icon('upload') + {{ trans('entities.import') }} + + @endif diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php new file mode 100644 index 000000000..df8f705cb --- /dev/null +++ b/resources/views/exports/import.blade.php @@ -0,0 +1,34 @@ +@extends('layouts.simple') + +@section('body') + +
    + +
    +
    +
    +

    {{ trans('entities.import') }}

    +

    + TODO - Desc +{{-- {{ trans('entities.permissions_desc') }}--}} +

    +
    +
    +
    + {{ csrf_field() }} +
    +
    + @include('form.checkbox', ['name' => 'images', 'label' => 'Include Images']) + @include('form.checkbox', ['name' => 'attachments', 'label' => 'Include Attachments']) +
    +
    + +
    + {{ trans('common.cancel') }} + +
    +
    +
    +
    + +@stop diff --git a/resources/views/settings/roles/parts/form.blade.php b/resources/views/settings/roles/parts/form.blade.php index 9fa76f2bf..a77b80e4c 100644 --- a/resources/views/settings/roles/parts/form.blade.php +++ b/resources/views/settings/roles/parts/form.blade.php @@ -37,6 +37,7 @@
    @include('settings.roles.parts.checkbox', ['permission' => 'templates-manage', 'label' => trans('settings.role_manage_page_templates')])
    @include('settings.roles.parts.checkbox', ['permission' => 'access-api', 'label' => trans('settings.role_access_api')])
    @include('settings.roles.parts.checkbox', ['permission' => 'content-export', 'label' => trans('settings.role_export_content')])
    +
    @include('settings.roles.parts.checkbox', ['permission' => 'content-import', 'label' => trans('settings.role_import_content')])
    @include('settings.roles.parts.checkbox', ['permission' => 'editor-change', 'label' => trans('settings.role_editor_change')])
    @include('settings.roles.parts.checkbox', ['permission' => 'receive-notifications', 'label' => trans('settings.role_notifications')])
    diff --git a/routes/web.php b/routes/web.php index e6f3683c6..91aab13fe 100644 --- a/routes/web.php +++ b/routes/web.php @@ -206,6 +206,10 @@ Route::middleware('auth')->group(function () { // Watching Route::put('/watching/update', [ActivityControllers\WatchController::class, 'update']); + // Importing + Route::get('/import', [ExportControllers\ImportController::class, 'start']); + Route::post('/import', [ExportControllers\ImportController::class, 'upload']); + // Other Pages Route::get('/', [HomeController::class, 'index']); Route::get('/home', [HomeController::class, 'index']); From a56a28fbb7eaff40a639c2d06f56de255cd654ea Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 29 Oct 2024 14:21:32 +0000 Subject: [PATCH 15/41] ZIP Exports: Built out initial import view Added syles for non-custom, non-image file inputs. Started planning out back-end handling. --- app/Exports/Controllers/ImportController.php | 8 ++++- lang/en/entities.php | 1 + resources/sass/_forms.scss | 37 ++++++++++++++++++++ resources/views/exports/import.blade.php | 31 ++++++++-------- 4 files changed, 62 insertions(+), 15 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index acc803a0f..9eefb0974 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -14,11 +14,17 @@ class ImportController extends Controller public function start(Request $request) { + // TODO - Show existing imports for user (or for all users if admin-level user) + return view('exports.import'); } public function upload(Request $request) { - // TODO + // TODO - Read existing ZIP upload and send through validator + // TODO - If invalid, return user with errors + // TODO - Upload to storage + // TODO - Store info/results from validator + // TODO - Send user to next import stage } } diff --git a/lang/en/entities.php b/lang/en/entities.php index 1a61b629a..45ca4cf6b 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -44,6 +44,7 @@ return [ 'default_template_explain' => 'Assign a page template that will be used as the default content for all pages created within this item. Keep in mind this will only be used if the page creator has view access to the chosen template page.', 'default_template_select' => 'Select a template page', 'import' => 'Import', + 'import_validate' => 'Validate Import', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/resources/sass/_forms.scss b/resources/sass/_forms.scss index 67df41714..1c679aaa0 100644 --- a/resources/sass/_forms.scss +++ b/resources/sass/_forms.scss @@ -545,6 +545,43 @@ input[type=color] { outline: 1px solid var(--color-primary); } +.custom-simple-file-input { + max-width: 100%; + border: 1px solid; + @include lightDark(border-color, #DDD, #666); + border-radius: 4px; + padding: $-s $-m; +} +.custom-simple-file-input::file-selector-button { + background-color: transparent; + text-decoration: none; + font-size: 0.8rem; + line-height: 1.4em; + padding: $-xs $-s; + border: 1px solid; + font-weight: 400; + outline: 0; + border-radius: 4px; + cursor: pointer; + margin-right: $-m; + @include lightDark(color, #666, #AAA); + @include lightDark(border-color, #CCC, #666); + &:hover, &:focus, &:active { + @include lightDark(color, #444, #BBB); + border: 1px solid #CCC; + box-shadow: 0 1px 4px 0 rgba(0, 0, 0, 0.1); + background-color: #F2F2F2; + @include lightDark(background-color, #f8f8f8, #444); + filter: none; + } + &:active { + border-color: #BBB; + background-color: #DDD; + color: #666; + box-shadow: inset 0 0 2px rgba(0, 0, 0, 0.1); + } +} + input.shortcut-input { width: auto; max-width: 120px; diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index df8f705cb..b7030f114 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -5,27 +5,30 @@
    -
    -
    -

    {{ trans('entities.import') }}

    -

    - TODO - Desc -{{-- {{ trans('entities.permissions_desc') }}--}} -

    -
    -
    +

    {{ trans('entities.import') }}

    {{ csrf_field() }} -
    -
    - @include('form.checkbox', ['name' => 'images', 'label' => 'Include Images']) - @include('form.checkbox', ['name' => 'attachments', 'label' => 'Include Attachments']) +
    +

    + Import content using a portable zip export from the same, or a different, instance. + Select a ZIP file to import then press "Validate Import" to proceed. + After the file has been uploaded and validated you'll be able to configure & confirm the import in the next view. +

    +
    +
    + + +
    {{ trans('common.cancel') }} - +
    From b50b7b667d2266950baa56457f2ed8b7eeda273d Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 30 Oct 2024 13:13:41 +0000 Subject: [PATCH 16/41] ZIP Exports: Started import validation --- .../ZipExportValidationException.php | 12 ++++ app/Exports/Controllers/ImportController.php | 6 ++ .../ZipExports/Models/ZipExportAttachment.php | 14 +++++ .../ZipExports/Models/ZipExportModel.php | 9 +++ .../ZipExports/Models/ZipExportTag.php | 12 ++++ app/Exports/ZipExports/ZipExportValidator.php | 63 +++++++++++++++++++ .../ZipExports/ZipFileReferenceRule.php | 26 ++++++++ .../ZipExports/ZipValidationHelper.php | 32 ++++++++++ lang/en/validation.php | 2 + resources/views/exports/import.blade.php | 2 +- 10 files changed, 177 insertions(+), 1 deletion(-) create mode 100644 app/Exceptions/ZipExportValidationException.php create mode 100644 app/Exports/ZipExports/ZipExportValidator.php create mode 100644 app/Exports/ZipExports/ZipFileReferenceRule.php create mode 100644 app/Exports/ZipExports/ZipValidationHelper.php diff --git a/app/Exceptions/ZipExportValidationException.php b/app/Exceptions/ZipExportValidationException.php new file mode 100644 index 000000000..2ed567d63 --- /dev/null +++ b/app/Exceptions/ZipExportValidationException.php @@ -0,0 +1,12 @@ +validate($request, [ + 'file' => ['required', 'file'] + ]); + + $file = $request->file('file'); + $file->getRealPath(); // TODO - Read existing ZIP upload and send through validator // TODO - If invalid, return user with errors // TODO - Upload to storage diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index 283ffa751..ab1f5ab75 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -3,6 +3,7 @@ namespace BookStack\Exports\ZipExports\Models; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; use BookStack\Uploads\Attachment; class ZipExportAttachment extends ZipExportModel @@ -35,4 +36,17 @@ class ZipExportAttachment extends ZipExportModel return self::fromModel($attachment, $files); }, $attachmentArray)); } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'order' => ['nullable', 'integer'], + 'link' => ['required_without:file', 'nullable', 'string'], + 'file' => ['required_without:link', 'nullable', 'string', $context->fileReferenceRule()], + ]; + + return $context->validateArray($data, $rules); + } } diff --git a/app/Exports/ZipExports/Models/ZipExportModel.php b/app/Exports/ZipExports/Models/ZipExportModel.php index 8d0c0a437..4d66f010f 100644 --- a/app/Exports/ZipExports/Models/ZipExportModel.php +++ b/app/Exports/ZipExports/Models/ZipExportModel.php @@ -2,6 +2,7 @@ namespace BookStack\Exports\ZipExports\Models; +use BookStack\Exports\ZipExports\ZipValidationHelper; use JsonSerializable; abstract class ZipExportModel implements JsonSerializable @@ -17,4 +18,12 @@ abstract class ZipExportModel implements JsonSerializable $publicProps = get_object_vars(...)->__invoke($this); return array_filter($publicProps, fn ($value) => $value !== null); } + + /** + * Validate the given array of data intended for this model. + * Return an array of validation errors messages. + * Child items can be considered in the validation result by returning a keyed + * item in the array for its own validation messages. + */ + abstract public static function validate(ZipValidationHelper $context, array $data): array; } diff --git a/app/Exports/ZipExports/Models/ZipExportTag.php b/app/Exports/ZipExports/Models/ZipExportTag.php index d4e3c4290..ad17d5a33 100644 --- a/app/Exports/ZipExports/Models/ZipExportTag.php +++ b/app/Exports/ZipExports/Models/ZipExportTag.php @@ -3,6 +3,7 @@ namespace BookStack\Exports\ZipExports\Models; use BookStack\Activity\Models\Tag; +use BookStack\Exports\ZipExports\ZipValidationHelper; class ZipExportTag extends ZipExportModel { @@ -24,4 +25,15 @@ class ZipExportTag extends ZipExportModel { return array_values(array_map(self::fromModel(...), $tagArray)); } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'name' => ['required', 'string', 'min:1'], + 'value' => ['nullable', 'string'], + 'order' => ['nullable', 'integer'], + ]; + + return $context->validateArray($data, $rules); + } } diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php new file mode 100644 index 000000000..5ad9272de --- /dev/null +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -0,0 +1,63 @@ +zipPath) || !is_readable($this->zipPath)) { + $this->throwErrors("Could not read ZIP file"); + } + + // Validate file is valid zip + $zip = new \ZipArchive(); + $opened = $zip->open($this->zipPath, ZipArchive::RDONLY); + if ($opened !== true) { + $this->throwErrors("Could not read ZIP file"); + } + + // Validate json data exists, including metadata + $jsonData = $zip->getFromName('data.json') ?: ''; + $importData = json_decode($jsonData, true); + if (!$importData) { + $this->throwErrors("Could not decode ZIP data.json content"); + } + + if (isset($importData['book'])) { + // TODO - Validate book + } else if (isset($importData['chapter'])) { + // TODO - Validate chapter + } else if (isset($importData['page'])) { + // TODO - Validate page + } else { + $this->throwErrors("ZIP file has no book, chapter or page data"); + } + } + + /** + * @throws ZipExportValidationException + */ + protected function throwErrors(...$errorsToAdd): never + { + array_push($this->errors, ...$errorsToAdd); + throw new ZipExportValidationException($this->errors); + } +} diff --git a/app/Exports/ZipExports/ZipFileReferenceRule.php b/app/Exports/ZipExports/ZipFileReferenceRule.php new file mode 100644 index 000000000..4f942e0e7 --- /dev/null +++ b/app/Exports/ZipExports/ZipFileReferenceRule.php @@ -0,0 +1,26 @@ +context->zipFileExists($value)) { + $fail('validation.zip_file')->translate(); + } + } +} diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php new file mode 100644 index 000000000..dd41e6f8b --- /dev/null +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -0,0 +1,32 @@ +validationFactory = app(Factory::class); + } + + public function validateArray(array $data, array $rules): array + { + return $this->validationFactory->make($data, $rules)->errors()->messages(); + } + + public function zipFileExists(string $name): bool + { + return $this->zip->statName("files/{$name}") !== false; + } + + public function fileReferenceRule(): ZipFileReferenceRule + { + return new ZipFileReferenceRule($this); + } +} diff --git a/lang/en/validation.php b/lang/en/validation.php index 2a676c7c4..6971edc02 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -105,6 +105,8 @@ return [ 'url' => 'The :attribute format is invalid.', 'uploaded' => 'The file could not be uploaded. The server may not accept files of this size.', + 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', + // Custom validation lines 'custom' => [ 'password-confirm' => [ diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index b7030f114..9fe596d88 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -10,7 +10,7 @@ {{ csrf_field() }}

    - Import content using a portable zip export from the same, or a different, instance. + Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to import then press "Validate Import" to proceed. After the file has been uploaded and validated you'll be able to configure & confirm the import in the next view.

    From c4ec50d437e52ccd831b6fb2e43baa5cf255fd1a Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 30 Oct 2024 15:26:23 +0000 Subject: [PATCH 17/41] ZIP Exports: Got zip format validation functionally complete --- .../ZipExportValidationException.php | 12 ----- app/Exports/Controllers/ImportController.php | 9 +++- .../ZipExports/Models/ZipExportAttachment.php | 2 +- .../ZipExports/Models/ZipExportBook.php | 21 ++++++++ .../ZipExports/Models/ZipExportChapter.php | 19 +++++++ .../ZipExports/Models/ZipExportImage.php | 14 +++++ .../ZipExports/Models/ZipExportPage.php | 22 ++++++++ .../ZipExports/Models/ZipExportTag.php | 2 +- app/Exports/ZipExports/ZipExportValidator.php | 53 +++++++++++-------- .../ZipExports/ZipValidationHelper.php | 31 ++++++++++- lang/en/validation.php | 3 +- resources/views/exports/import.blade.php | 3 +- 12 files changed, 149 insertions(+), 42 deletions(-) delete mode 100644 app/Exceptions/ZipExportValidationException.php diff --git a/app/Exceptions/ZipExportValidationException.php b/app/Exceptions/ZipExportValidationException.php deleted file mode 100644 index 2ed567d63..000000000 --- a/app/Exceptions/ZipExportValidationException.php +++ /dev/null @@ -1,12 +0,0 @@ -file('file'); - $file->getRealPath(); + $zipPath = $file->getRealPath(); + + $errors = (new ZipExportValidator($zipPath))->validate(); + if ($errors) { + dd($errors); + } + dd('passed'); // TODO - Read existing ZIP upload and send through validator // TODO - If invalid, return user with errors // TODO - Upload to storage diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index ab1f5ab75..e586b91b0 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -47,6 +47,6 @@ class ZipExportAttachment extends ZipExportModel 'file' => ['required_without:link', 'nullable', 'string', $context->fileReferenceRule()], ]; - return $context->validateArray($data, $rules); + return $context->validateData($data, $rules); } } diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php index 5a0c5806b..7e1f2d810 100644 --- a/app/Exports/ZipExports/Models/ZipExportBook.php +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -6,6 +6,7 @@ use BookStack\Entities\Models\Book; use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; class ZipExportBook extends ZipExportModel { @@ -50,4 +51,24 @@ class ZipExportBook extends ZipExportModel return $instance; } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'description_html' => ['nullable', 'string'], + 'cover' => ['nullable', 'string', $context->fileReferenceRule()], + 'tags' => ['array'], + 'pages' => ['array'], + 'chapters' => ['array'], + ]; + + $errors = $context->validateData($data, $rules); + $errors['tags'] = $context->validateRelations($data['tags'] ?? [], ZipExportTag::class); + $errors['pages'] = $context->validateRelations($data['pages'] ?? [], ZipExportPage::class); + $errors['chapters'] = $context->validateRelations($data['chapters'] ?? [], ZipExportChapter::class); + + return $errors; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php index cd5765f48..03df31b70 100644 --- a/app/Exports/ZipExports/Models/ZipExportChapter.php +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -5,6 +5,7 @@ namespace BookStack\Exports\ZipExports\Models; use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; class ZipExportChapter extends ZipExportModel { @@ -42,4 +43,22 @@ class ZipExportChapter extends ZipExportModel return self::fromModel($chapter, $files); }, $chapterArray)); } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'description_html' => ['nullable', 'string'], + 'priority' => ['nullable', 'int'], + 'tags' => ['array'], + 'pages' => ['array'], + ]; + + $errors = $context->validateData($data, $rules); + $errors['tags'] = $context->validateRelations($data['tags'] ?? [], ZipExportTag::class); + $errors['pages'] = $context->validateRelations($data['pages'] ?? [], ZipExportPage::class); + + return $errors; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportImage.php b/app/Exports/ZipExports/Models/ZipExportImage.php index 05d828734..3388c66df 100644 --- a/app/Exports/ZipExports/Models/ZipExportImage.php +++ b/app/Exports/ZipExports/Models/ZipExportImage.php @@ -3,7 +3,9 @@ namespace BookStack\Exports\ZipExports\Models; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; use BookStack\Uploads\Image; +use Illuminate\Validation\Rule; class ZipExportImage extends ZipExportModel { @@ -22,4 +24,16 @@ class ZipExportImage extends ZipExportModel return $instance; } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'file' => ['required', 'string', $context->fileReferenceRule()], + 'type' => ['required', 'string', Rule::in(['gallery', 'drawio'])], + ]; + + return $context->validateData($data, $rules); + } } diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index 8075595f2..2c8b9a88a 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -5,6 +5,7 @@ namespace BookStack\Exports\ZipExports\Models; use BookStack\Entities\Models\Page; use BookStack\Entities\Tools\PageContent; use BookStack\Exports\ZipExports\ZipExportFiles; +use BookStack\Exports\ZipExports\ZipValidationHelper; class ZipExportPage extends ZipExportModel { @@ -48,4 +49,25 @@ class ZipExportPage extends ZipExportModel return self::fromModel($page, $files); }, $pageArray)); } + + public static function validate(ZipValidationHelper $context, array $data): array + { + $rules = [ + 'id' => ['nullable', 'int'], + 'name' => ['required', 'string', 'min:1'], + 'html' => ['nullable', 'string'], + 'markdown' => ['nullable', 'string'], + 'priority' => ['nullable', 'int'], + 'attachments' => ['array'], + 'images' => ['array'], + 'tags' => ['array'], + ]; + + $errors = $context->validateData($data, $rules); + $errors['attachments'] = $context->validateRelations($data['attachments'] ?? [], ZipExportAttachment::class); + $errors['images'] = $context->validateRelations($data['images'] ?? [], ZipExportImage::class); + $errors['tags'] = $context->validateRelations($data['tags'] ?? [], ZipExportTag::class); + + return $errors; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportTag.php b/app/Exports/ZipExports/Models/ZipExportTag.php index ad17d5a33..99abb811c 100644 --- a/app/Exports/ZipExports/Models/ZipExportTag.php +++ b/app/Exports/ZipExports/Models/ZipExportTag.php @@ -34,6 +34,6 @@ class ZipExportTag extends ZipExportModel 'order' => ['nullable', 'integer'], ]; - return $context->validateArray($data, $rules); + return $context->validateData($data, $rules); } } diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index 5ad9272de..e56394aca 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -2,62 +2,69 @@ namespace BookStack\Exports\ZipExports; -use BookStack\Exceptions\ZipExportValidationException; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use ZipArchive; class ZipExportValidator { - protected array $errors = []; - public function __construct( protected string $zipPath, ) { } - /** - * @throws ZipExportValidationException - */ - public function validate() + public function validate(): array { - // TODO - Return type - // TODO - extract messages to translations? - // Validate file exists if (!file_exists($this->zipPath) || !is_readable($this->zipPath)) { - $this->throwErrors("Could not read ZIP file"); + return ['format' => "Could not read ZIP file"]; } // Validate file is valid zip $zip = new \ZipArchive(); $opened = $zip->open($this->zipPath, ZipArchive::RDONLY); if ($opened !== true) { - $this->throwErrors("Could not read ZIP file"); + return ['format' => "Could not read ZIP file"]; } // Validate json data exists, including metadata $jsonData = $zip->getFromName('data.json') ?: ''; $importData = json_decode($jsonData, true); if (!$importData) { - $this->throwErrors("Could not decode ZIP data.json content"); + return ['format' => "Could not find and decode ZIP data.json content"]; } + $helper = new ZipValidationHelper($zip); + if (isset($importData['book'])) { - // TODO - Validate book + $modelErrors = ZipExportBook::validate($helper, $importData['book']); + $keyPrefix = 'book'; } else if (isset($importData['chapter'])) { - // TODO - Validate chapter + $modelErrors = ZipExportChapter::validate($helper, $importData['chapter']); + $keyPrefix = 'chapter'; } else if (isset($importData['page'])) { - // TODO - Validate page + $modelErrors = ZipExportPage::validate($helper, $importData['page']); + $keyPrefix = 'page'; } else { - $this->throwErrors("ZIP file has no book, chapter or page data"); + return ['format' => "ZIP file has no book, chapter or page data"]; } + + return $this->flattenModelErrors($modelErrors, $keyPrefix); } - /** - * @throws ZipExportValidationException - */ - protected function throwErrors(...$errorsToAdd): never + protected function flattenModelErrors(array $errors, string $keyPrefix): array { - array_push($this->errors, ...$errorsToAdd); - throw new ZipExportValidationException($this->errors); + $flattened = []; + + foreach ($errors as $key => $error) { + if (is_array($error)) { + $flattened = array_merge($flattened, $this->flattenModelErrors($error, $keyPrefix . '.' . $key)); + } else { + $flattened[$keyPrefix . '.' . $key] = $error; + } + } + + return $flattened; } } diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php index dd41e6f8b..8c285deaf 100644 --- a/app/Exports/ZipExports/ZipValidationHelper.php +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -2,6 +2,7 @@ namespace BookStack\Exports\ZipExports; +use BookStack\Exports\ZipExports\Models\ZipExportModel; use Illuminate\Validation\Factory; use ZipArchive; @@ -15,9 +16,15 @@ class ZipValidationHelper $this->validationFactory = app(Factory::class); } - public function validateArray(array $data, array $rules): array + public function validateData(array $data, array $rules): array { - return $this->validationFactory->make($data, $rules)->errors()->messages(); + $messages = $this->validationFactory->make($data, $rules)->errors()->messages(); + + foreach ($messages as $key => $message) { + $messages[$key] = implode("\n", $message); + } + + return $messages; } public function zipFileExists(string $name): bool @@ -29,4 +36,24 @@ class ZipValidationHelper { return new ZipFileReferenceRule($this); } + + /** + * Validate an array of relation data arrays that are expected + * to be for the given ZipExportModel. + * @param class-string $model + */ + public function validateRelations(array $relations, string $model): array + { + $results = []; + + foreach ($relations as $key => $relationData) { + if (is_array($relationData)) { + $results[$key] = $model::validate($this, $relationData); + } else { + $results[$key] = [trans('validation.zip_model_expected', ['type' => gettype($relationData)])]; + } + } + + return $results; + } } diff --git a/lang/en/validation.php b/lang/en/validation.php index 6971edc02..9cf5d78b6 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -105,7 +105,8 @@ return [ 'url' => 'The :attribute format is invalid.', 'uploaded' => 'The file could not be uploaded. The server may not accept files of this size.', - 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', + 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', + 'zip_model_expected' => 'Data object expected but ":type" found', // Custom validation lines 'custom' => [ diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index 9fe596d88..15f33e6b7 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -6,7 +6,7 @@

    {{ trans('entities.import') }}

    -
    + {{ csrf_field() }}

    @@ -22,6 +22,7 @@ name="file" id="file" class="custom-simple-file-input"> + @include('form.errors', ['name' => 'file'])

    From 259aa829d42b1cd93011d5b8b531c15804741cb5 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 2 Nov 2024 14:51:04 +0000 Subject: [PATCH 18/41] ZIP Imports: Added validation message display, added testing Testing covers main UI access, and main non-successfull import actions. Started planning stored import model. Extracted some text to language files. --- app/Exports/Controllers/ImportController.php | 20 ++- app/Exports/ZipExports/ZipExportValidator.php | 9 +- lang/en/entities.php | 3 + lang/en/errors.php | 5 + lang/en/validation.php | 2 +- resources/views/exports/import.blade.php | 17 ++- tests/Exports/ZipImportTest.php | 124 ++++++++++++++++++ 7 files changed, 164 insertions(+), 16 deletions(-) create mode 100644 tests/Exports/ZipImportTest.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 5885f7991..323ecef26 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -17,7 +17,9 @@ class ImportController extends Controller { // TODO - Show existing imports for user (or for all users if admin-level user) - return view('exports.import'); + return view('exports.import', [ + 'zipErrors' => session()->pull('validation_errors') ?? [], + ]); } public function upload(Request $request) @@ -31,13 +33,21 @@ class ImportController extends Controller $errors = (new ZipExportValidator($zipPath))->validate(); if ($errors) { - dd($errors); + session()->flash('validation_errors', $errors); + return redirect('/import'); } + dd('passed'); - // TODO - Read existing ZIP upload and send through validator - // TODO - If invalid, return user with errors // TODO - Upload to storage - // TODO - Store info/results from validator + // TODO - Store info/results for display: + // - zip_path + // - name (From name of thing being imported) + // - size + // - book_count + // - chapter_count + // - page_count + // - created_by + // - created_at/updated_at // TODO - Send user to next import stage } } diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index e56394aca..dd56f3e70 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -18,21 +18,21 @@ class ZipExportValidator { // Validate file exists if (!file_exists($this->zipPath) || !is_readable($this->zipPath)) { - return ['format' => "Could not read ZIP file"]; + return ['format' => trans('errors.import_zip_cant_read')]; } // Validate file is valid zip $zip = new \ZipArchive(); $opened = $zip->open($this->zipPath, ZipArchive::RDONLY); if ($opened !== true) { - return ['format' => "Could not read ZIP file"]; + return ['format' => trans('errors.import_zip_cant_read')]; } // Validate json data exists, including metadata $jsonData = $zip->getFromName('data.json') ?: ''; $importData = json_decode($jsonData, true); if (!$importData) { - return ['format' => "Could not find and decode ZIP data.json content"]; + return ['format' => trans('errors.import_zip_cant_decode_data')]; } $helper = new ZipValidationHelper($zip); @@ -47,9 +47,10 @@ class ZipExportValidator $modelErrors = ZipExportPage::validate($helper, $importData['page']); $keyPrefix = 'page'; } else { - return ['format' => "ZIP file has no book, chapter or page data"]; + return ['format' => trans('errors.import_zip_no_data')]; } + return $this->flattenModelErrors($modelErrors, $keyPrefix); } diff --git a/lang/en/entities.php b/lang/en/entities.php index 45ca4cf6b..106147335 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -45,6 +45,9 @@ return [ 'default_template_select' => 'Select a template page', 'import' => 'Import', 'import_validate' => 'Validate Import', + 'import_desc' => 'Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to import then press "Validate Import" to proceed. After the file has been uploaded and validated you\'ll be able to configure & confirm the import in the next view.', + 'import_zip_select' => 'Select ZIP file to upload', + 'import_zip_validation_errors' => 'Errors were detected while validating the provided ZIP file:', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/lang/en/errors.php b/lang/en/errors.php index 9c40aa9ed..3f2f30331 100644 --- a/lang/en/errors.php +++ b/lang/en/errors.php @@ -105,6 +105,11 @@ return [ 'app_down' => ':appName is down right now', 'back_soon' => 'It will be back up soon.', + // Import + 'import_zip_cant_read' => 'Could not read ZIP file.', + 'import_zip_cant_decode_data' => 'Could not find and decode ZIP data.json content.', + 'import_zip_no_data' => 'ZIP file data has no expected book, chapter or page content.', + // API errors 'api_no_authorization_found' => 'No authorization token found on the request', 'api_bad_authorization_format' => 'An authorization token was found on the request but the format appeared incorrect', diff --git a/lang/en/validation.php b/lang/en/validation.php index 9cf5d78b6..bc01ac47b 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -106,7 +106,7 @@ return [ 'uploaded' => 'The file could not be uploaded. The server may not accept files of this size.', 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', - 'zip_model_expected' => 'Data object expected but ":type" found', + 'zip_model_expected' => 'Data object expected but ":type" found.', // Custom validation lines 'custom' => [ diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index 15f33e6b7..c4d7c8818 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -9,14 +9,10 @@ {{ csrf_field() }}
    -

    - Import books, chapters & pages using a portable zip export from the same, or a different, instance. - Select a ZIP file to import then press "Validate Import" to proceed. - After the file has been uploaded and validated you'll be able to configure & confirm the import in the next view. -

    +

    {{ trans('entities.import_desc') }}

    - +
    + @if(count($zipErrors) > 0) +

    {{ trans('entities.import_zip_validation_errors') }}

    +
      + @foreach($zipErrors as $key => $error) +
    • [{{ $key }}]: {{ $error }}
      • + @endforeach +
    + @endif +
    {{ trans('common.cancel') }} diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php new file mode 100644 index 000000000..c9d255b1e --- /dev/null +++ b/tests/Exports/ZipImportTest.php @@ -0,0 +1,124 @@ +asAdmin()->get('/import'); + $resp->assertSee('Import'); + $this->withHtml($resp)->assertElementExists('form input[type="file"][name="file"]'); + } + + public function test_permissions_needed_for_import_page() + { + $user = $this->users->viewer(); + $this->actingAs($user); + + $resp = $this->get('/books'); + $this->withHtml($resp)->assertLinkNotExists(url('/import')); + $resp = $this->get('/import'); + $resp->assertRedirect('/'); + + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $resp = $this->get('/books'); + $this->withHtml($resp)->assertLinkExists(url('/import')); + $resp = $this->get('/import'); + $resp->assertOk(); + $resp->assertSeeText('Select ZIP file to upload'); + } + + public function test_zip_read_errors_are_shown_on_validation() + { + $invalidUpload = $this->files->uploadedImage('image.zip'); + + $this->asAdmin(); + $resp = $this->runImportFromFile($invalidUpload); + $resp->assertRedirect('/import'); + + $resp = $this->followRedirects($resp); + $resp->assertSeeText('Could not read ZIP file'); + } + + public function test_error_shown_if_missing_data() + { + $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::CREATE); + $zip->addFromString('beans', 'cat'); + $zip->close(); + + $this->asAdmin(); + $upload = new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); + $resp = $this->runImportFromFile($upload); + $resp->assertRedirect('/import'); + + $resp = $this->followRedirects($resp); + $resp->assertSeeText('Could not find and decode ZIP data.json content.'); + } + + public function test_error_shown_if_no_importable_key() + { + $this->asAdmin(); + $resp = $this->runImportFromFile($this->zipUploadFromData([ + 'instance' => [] + ])); + + $resp->assertRedirect('/import'); + $resp = $this->followRedirects($resp); + $resp->assertSeeText('ZIP file data has no expected book, chapter or page content.'); + } + + public function test_zip_data_validation_messages_shown() + { + $this->asAdmin(); + $resp = $this->runImportFromFile($this->zipUploadFromData([ + 'book' => [ + 'id' => 4, + 'pages' => [ + 'cat', + [ + 'name' => 'My inner page', + 'tags' => [ + [ + 'value' => 5 + ] + ], + ] + ], + ] + ])); + + $resp->assertRedirect('/import'); + $resp = $this->followRedirects($resp); + + $resp->assertSeeText('[book.name]: The name field is required.'); + $resp->assertSeeText('[book.pages.0.0]: Data object expected but "string" found.'); + $resp->assertSeeText('[book.pages.1.tags.0.name]: The name field is required.'); + $resp->assertSeeText('[book.pages.1.tags.0.value]: The value must be a string.'); + } + + protected function runImportFromFile(UploadedFile $file): TestResponse + { + return $this->call('POST', '/import', [], [], ['file' => $file]); + } + + protected function zipUploadFromData(array $data): UploadedFile + { + $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); + + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::CREATE); + $zip->addFromString('data.json', json_encode($data)); + $zip->close(); + + return new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); + } +} From 74fce9640ef39a743bdb5a997724465c7c2b764c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 2 Nov 2024 17:17:34 +0000 Subject: [PATCH 19/41] ZIP Import: Added model+migration, and reader class --- app/Exports/Controllers/ImportController.php | 24 +++-- app/Exports/Import.php | 41 +++++++ app/Exports/ZipExports/ZipExportReader.php | 102 ++++++++++++++++++ app/Exports/ZipExports/ZipExportValidator.php | 26 ++--- .../ZipExports/ZipFileReferenceRule.php | 2 +- .../ZipExports/ZipValidationHelper.php | 8 +- database/factories/Exports/ImportFactory.php | 32 ++++++ ...2024_11_02_160700_create_imports_table.php | 34 ++++++ 8 files changed, 234 insertions(+), 35 deletions(-) create mode 100644 app/Exports/Import.php create mode 100644 app/Exports/ZipExports/ZipExportReader.php create mode 100644 database/factories/Exports/ImportFactory.php create mode 100644 database/migrations/2024_11_02_160700_create_imports_table.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 323ecef26..bbf0ff57d 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -2,6 +2,8 @@ namespace BookStack\Exports\Controllers; +use BookStack\Exports\Import; +use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; use BookStack\Http\Controller; use Illuminate\Http\Request; @@ -37,17 +39,23 @@ class ImportController extends Controller return redirect('/import'); } + $zipEntityInfo = (new ZipExportReader($zipPath))->getEntityInfo(); + $import = new Import(); + $import->name = $zipEntityInfo['name']; + $import->book_count = $zipEntityInfo['book_count']; + $import->chapter_count = $zipEntityInfo['chapter_count']; + $import->page_count = $zipEntityInfo['page_count']; + $import->created_by = user()->id; + $import->size = filesize($zipPath); + // TODO - Set path + // TODO - Save + + // TODO - Split out attachment service to separate out core filesystem/disk stuff + // To reuse for import handling + dd('passed'); // TODO - Upload to storage // TODO - Store info/results for display: - // - zip_path - // - name (From name of thing being imported) - // - size - // - book_count - // - chapter_count - // - page_count - // - created_by - // - created_at/updated_at // TODO - Send user to next import stage } } diff --git a/app/Exports/Import.php b/app/Exports/Import.php new file mode 100644 index 000000000..c3ac3d529 --- /dev/null +++ b/app/Exports/Import.php @@ -0,0 +1,41 @@ +book_count === 1) { + return self::TYPE_BOOK; + } elseif ($this->chapter_count === 1) { + return self::TYPE_CHAPTER; + } + + return self::TYPE_PAGE; + } +} diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php new file mode 100644 index 000000000..7187a1889 --- /dev/null +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -0,0 +1,102 @@ +zip = new ZipArchive(); + } + + /** + * @throws ZipExportException + */ + protected function open(): void + { + if ($this->open) { + return; + } + + // Validate file exists + if (!file_exists($this->zipPath) || !is_readable($this->zipPath)) { + throw new ZipExportException(trans('errors.import_zip_cant_read')); + } + + // Validate file is valid zip + $opened = $this->zip->open($this->zipPath, ZipArchive::RDONLY); + if ($opened !== true) { + throw new ZipExportException(trans('errors.import_zip_cant_read')); + } + + $this->open = true; + } + + public function close(): void + { + if ($this->open) { + $this->zip->close(); + $this->open = false; + } + } + + /** + * @throws ZipExportException + */ + public function readData(): array + { + $this->open(); + + // Validate json data exists, including metadata + $jsonData = $this->zip->getFromName('data.json') ?: ''; + $importData = json_decode($jsonData, true); + if (!$importData) { + throw new ZipExportException(trans('errors.import_zip_cant_decode_data')); + } + + return $importData; + } + + public function fileExists(string $fileName): bool + { + return $this->zip->statName("files/{$fileName}") !== false; + } + + /** + * @throws ZipExportException + * @returns array{name: string, book_count: int, chapter_count: int, page_count: int} + */ + public function getEntityInfo(): array + { + $data = $this->readData(); + $info = ['name' => '', 'book_count' => 0, 'chapter_count' => 0, 'page_count' => 0]; + + if (isset($data['book'])) { + $info['name'] = $data['book']['name'] ?? ''; + $info['book_count']++; + $chapters = $data['book']['chapters'] ?? []; + $pages = $data['book']['pages'] ?? []; + $info['chapter_count'] += count($chapters); + $info['page_count'] += count($pages); + foreach ($chapters as $chapter) { + $info['page_count'] += count($chapter['pages'] ?? []); + } + } elseif (isset($data['chapter'])) { + $info['name'] = $data['chapter']['name'] ?? ''; + $info['chapter_count']++; + $info['page_count'] += count($data['chapter']['pages'] ?? []); + } elseif (isset($data['page'])) { + $info['name'] = $data['page']['name'] ?? ''; + $info['page_count']++; + } + + return $info; + } +} diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index dd56f3e70..e476998c2 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -2,10 +2,10 @@ namespace BookStack\Exports\ZipExports; +use BookStack\Exceptions\ZipExportException; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportPage; -use ZipArchive; class ZipExportValidator { @@ -16,26 +16,14 @@ class ZipExportValidator public function validate(): array { - // Validate file exists - if (!file_exists($this->zipPath) || !is_readable($this->zipPath)) { - return ['format' => trans('errors.import_zip_cant_read')]; + $reader = new ZipExportReader($this->zipPath); + try { + $importData = $reader->readData(); + } catch (ZipExportException $exception) { + return ['format' => $exception->getMessage()]; } - // Validate file is valid zip - $zip = new \ZipArchive(); - $opened = $zip->open($this->zipPath, ZipArchive::RDONLY); - if ($opened !== true) { - return ['format' => trans('errors.import_zip_cant_read')]; - } - - // Validate json data exists, including metadata - $jsonData = $zip->getFromName('data.json') ?: ''; - $importData = json_decode($jsonData, true); - if (!$importData) { - return ['format' => trans('errors.import_zip_cant_decode_data')]; - } - - $helper = new ZipValidationHelper($zip); + $helper = new ZipValidationHelper($reader); if (isset($importData['book'])) { $modelErrors = ZipExportBook::validate($helper, $importData['book']); diff --git a/app/Exports/ZipExports/ZipFileReferenceRule.php b/app/Exports/ZipExports/ZipFileReferenceRule.php index 4f942e0e7..bcd3c39ac 100644 --- a/app/Exports/ZipExports/ZipFileReferenceRule.php +++ b/app/Exports/ZipExports/ZipFileReferenceRule.php @@ -19,7 +19,7 @@ class ZipFileReferenceRule implements ValidationRule */ public function validate(string $attribute, mixed $value, Closure $fail): void { - if (!$this->context->zipFileExists($value)) { + if (!$this->context->zipReader->fileExists($value)) { $fail('validation.zip_file')->translate(); } } diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php index 8c285deaf..55c86b03b 100644 --- a/app/Exports/ZipExports/ZipValidationHelper.php +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -4,14 +4,13 @@ namespace BookStack\Exports\ZipExports; use BookStack\Exports\ZipExports\Models\ZipExportModel; use Illuminate\Validation\Factory; -use ZipArchive; class ZipValidationHelper { protected Factory $validationFactory; public function __construct( - protected ZipArchive $zip, + public ZipExportReader $zipReader, ) { $this->validationFactory = app(Factory::class); } @@ -27,11 +26,6 @@ class ZipValidationHelper return $messages; } - public function zipFileExists(string $name): bool - { - return $this->zip->statName("files/{$name}") !== false; - } - public function fileReferenceRule(): ZipFileReferenceRule { return new ZipFileReferenceRule($this); diff --git a/database/factories/Exports/ImportFactory.php b/database/factories/Exports/ImportFactory.php new file mode 100644 index 000000000..55378d583 --- /dev/null +++ b/database/factories/Exports/ImportFactory.php @@ -0,0 +1,32 @@ + 'uploads/imports/' . Str::random(10) . '.zip', + 'name' => $this->faker->words(3, true), + 'book_count' => 1, + 'chapter_count' => 5, + 'page_count' => 15, + 'created_at' => User::factory(), + ]; + } +} diff --git a/database/migrations/2024_11_02_160700_create_imports_table.php b/database/migrations/2024_11_02_160700_create_imports_table.php new file mode 100644 index 000000000..ed1882269 --- /dev/null +++ b/database/migrations/2024_11_02_160700_create_imports_table.php @@ -0,0 +1,34 @@ +increments('id'); + $table->string('name'); + $table->string('path'); + $table->integer('size'); + $table->integer('book_count'); + $table->integer('chapter_count'); + $table->integer('page_count'); + $table->integer('created_by'); + $table->timestamps(); + }); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + Schema::dropIfExists('imports'); + } +}; From 8ea3855e02aa5ff7782dc65e1eee8b8b24f28ce6 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 2 Nov 2024 20:48:21 +0000 Subject: [PATCH 20/41] ZIP Import: Added upload handling Split attachment service storage work out so it can be shared. --- app/Exceptions/ZipValidationException.php | 12 ++ app/Exports/Controllers/ImportController.php | 41 ++----- app/Exports/ImportRepo.php | 48 ++++++++ app/Uploads/AttachmentService.php | 86 ++------------ app/Uploads/FileStorage.php | 111 +++++++++++++++++++ 5 files changed, 195 insertions(+), 103 deletions(-) create mode 100644 app/Exceptions/ZipValidationException.php create mode 100644 app/Exports/ImportRepo.php create mode 100644 app/Uploads/FileStorage.php diff --git a/app/Exceptions/ZipValidationException.php b/app/Exceptions/ZipValidationException.php new file mode 100644 index 000000000..aaaee792e --- /dev/null +++ b/app/Exceptions/ZipValidationException.php @@ -0,0 +1,12 @@ +middleware('can:content-import'); } @@ -27,35 +28,17 @@ class ImportController extends Controller public function upload(Request $request) { $this->validate($request, [ - 'file' => ['required', 'file'] + 'file' => ['required', ...AttachmentService::getFileValidationRules()] ]); $file = $request->file('file'); - $zipPath = $file->getRealPath(); - - $errors = (new ZipExportValidator($zipPath))->validate(); - if ($errors) { - session()->flash('validation_errors', $errors); + try { + $import = $this->imports->storeFromUpload($file); + } catch (ZipValidationException $exception) { + session()->flash('validation_errors', $exception->errors); return redirect('/import'); } - $zipEntityInfo = (new ZipExportReader($zipPath))->getEntityInfo(); - $import = new Import(); - $import->name = $zipEntityInfo['name']; - $import->book_count = $zipEntityInfo['book_count']; - $import->chapter_count = $zipEntityInfo['chapter_count']; - $import->page_count = $zipEntityInfo['page_count']; - $import->created_by = user()->id; - $import->size = filesize($zipPath); - // TODO - Set path - // TODO - Save - - // TODO - Split out attachment service to separate out core filesystem/disk stuff - // To reuse for import handling - - dd('passed'); - // TODO - Upload to storage - // TODO - Store info/results for display: - // TODO - Send user to next import stage + return redirect("imports/{$import->id}"); } } diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php new file mode 100644 index 000000000..c8157967b --- /dev/null +++ b/app/Exports/ImportRepo.php @@ -0,0 +1,48 @@ +getRealPath(); + + $errors = (new ZipExportValidator($zipPath))->validate(); + if ($errors) { + throw new ZipValidationException($errors); + } + + $zipEntityInfo = (new ZipExportReader($zipPath))->getEntityInfo(); + $import = new Import(); + $import->name = $zipEntityInfo['name']; + $import->book_count = $zipEntityInfo['book_count']; + $import->chapter_count = $zipEntityInfo['chapter_count']; + $import->page_count = $zipEntityInfo['page_count']; + $import->created_by = user()->id; + $import->size = filesize($zipPath); + + $path = $this->storage->uploadFile( + $file, + 'uploads/files/imports/', + '', + 'zip' + ); + + $import->path = $path; + $import->save(); + + return $import; + } +} diff --git a/app/Uploads/AttachmentService.php b/app/Uploads/AttachmentService.php index 227649d8f..fa53c4ae4 100644 --- a/app/Uploads/AttachmentService.php +++ b/app/Uploads/AttachmentService.php @@ -4,59 +4,15 @@ namespace BookStack\Uploads; use BookStack\Exceptions\FileUploadException; use Exception; -use Illuminate\Contracts\Filesystem\Filesystem as Storage; -use Illuminate\Filesystem\FilesystemManager; -use Illuminate\Support\Facades\Log; -use Illuminate\Support\Str; -use League\Flysystem\WhitespacePathNormalizer; use Symfony\Component\HttpFoundation\File\UploadedFile; class AttachmentService { public function __construct( - protected FilesystemManager $fileSystem + protected FileStorage $storage, ) { } - /** - * Get the storage that will be used for storing files. - */ - protected function getStorageDisk(): Storage - { - return $this->fileSystem->disk($this->getStorageDiskName()); - } - - /** - * Get the name of the storage disk to use. - */ - protected function getStorageDiskName(): string - { - $storageType = config('filesystems.attachments'); - - // Change to our secure-attachment disk if any of the local options - // are used to prevent escaping that location. - if ($storageType === 'local' || $storageType === 'local_secure' || $storageType === 'local_secure_restricted') { - $storageType = 'local_secure_attachments'; - } - - return $storageType; - } - - /** - * Change the originally provided path to fit any disk-specific requirements. - * This also ensures the path is kept to the expected root folders. - */ - protected function adjustPathForStorageDisk(string $path): string - { - $path = (new WhitespacePathNormalizer())->normalizePath(str_replace('uploads/files/', '', $path)); - - if ($this->getStorageDiskName() === 'local_secure_attachments') { - return $path; - } - - return 'uploads/files/' . $path; - } - /** * Stream an attachment from storage. * @@ -64,7 +20,7 @@ class AttachmentService */ public function streamAttachmentFromStorage(Attachment $attachment) { - return $this->getStorageDisk()->readStream($this->adjustPathForStorageDisk($attachment->path)); + return $this->storage->getReadStream($attachment->path); } /** @@ -72,7 +28,7 @@ class AttachmentService */ public function getAttachmentFileSize(Attachment $attachment): int { - return $this->getStorageDisk()->size($this->adjustPathForStorageDisk($attachment->path)); + return $this->storage->getSize($attachment->path); } /** @@ -195,15 +151,9 @@ class AttachmentService * Delete a file from the filesystem it sits on. * Cleans any empty leftover folders. */ - protected function deleteFileInStorage(Attachment $attachment) + protected function deleteFileInStorage(Attachment $attachment): void { - $storage = $this->getStorageDisk(); - $dirPath = $this->adjustPathForStorageDisk(dirname($attachment->path)); - - $storage->delete($this->adjustPathForStorageDisk($attachment->path)); - if (count($storage->allFiles($dirPath)) === 0) { - $storage->deleteDirectory($dirPath); - } + $this->storage->delete($attachment->path); } /** @@ -213,32 +163,20 @@ class AttachmentService */ protected function putFileInStorage(UploadedFile $uploadedFile): string { - $storage = $this->getStorageDisk(); $basePath = 'uploads/files/' . date('Y-m-M') . '/'; - $uploadFileName = Str::random(16) . '-' . $uploadedFile->getClientOriginalExtension(); - while ($storage->exists($this->adjustPathForStorageDisk($basePath . $uploadFileName))) { - $uploadFileName = Str::random(3) . $uploadFileName; - } - - $attachmentStream = fopen($uploadedFile->getRealPath(), 'r'); - $attachmentPath = $basePath . $uploadFileName; - - try { - $storage->writeStream($this->adjustPathForStorageDisk($attachmentPath), $attachmentStream); - } catch (Exception $e) { - Log::error('Error when attempting file upload:' . $e->getMessage()); - - throw new FileUploadException(trans('errors.path_not_writable', ['filePath' => $attachmentPath])); - } - - return $attachmentPath; + return $this->storage->uploadFile( + $uploadedFile, + $basePath, + $uploadedFile->getClientOriginalExtension(), + '' + ); } /** * Get the file validation rules for attachments. */ - public function getFileValidationRules(): array + public static function getFileValidationRules(): array { return ['file', 'max:' . (config('app.upload_limit') * 1000)]; } diff --git a/app/Uploads/FileStorage.php b/app/Uploads/FileStorage.php new file mode 100644 index 000000000..278484e51 --- /dev/null +++ b/app/Uploads/FileStorage.php @@ -0,0 +1,111 @@ +getStorageDisk()->readStream($this->adjustPathForStorageDisk($path)); + } + + public function getSize(string $path): int + { + return $this->getStorageDisk()->size($this->adjustPathForStorageDisk($path)); + } + + public function delete(string $path, bool $removeEmptyDir = false): void + { + $storage = $this->getStorageDisk(); + $adjustedPath = $this->adjustPathForStorageDisk($path); + $dir = dirname($adjustedPath); + + $storage->delete($adjustedPath); + if ($removeEmptyDir && count($storage->allFiles($dir)) === 0) { + $storage->deleteDirectory($dir); + } + } + + /** + * @throws FileUploadException + */ + public function uploadFile(UploadedFile $file, string $subDirectory, string $suffix, string $extension): string + { + $storage = $this->getStorageDisk(); + $basePath = trim($subDirectory, '/') . '/'; + + $uploadFileName = Str::random(16) . ($suffix ? "-{$suffix}" : '') . ($extension ? ".{$extension}" : ''); + while ($storage->exists($this->adjustPathForStorageDisk($basePath . $uploadFileName))) { + $uploadFileName = Str::random(3) . $uploadFileName; + } + + $fileStream = fopen($file->getRealPath(), 'r'); + $filePath = $basePath . $uploadFileName; + + try { + $storage->writeStream($this->adjustPathForStorageDisk($filePath), $fileStream); + } catch (Exception $e) { + Log::error('Error when attempting file upload:' . $e->getMessage()); + + throw new FileUploadException(trans('errors.path_not_writable', ['filePath' => $filePath])); + } + + return $filePath; + } + + /** + * Get the storage that will be used for storing files. + */ + protected function getStorageDisk(): Storage + { + return $this->fileSystem->disk($this->getStorageDiskName()); + } + + /** + * Get the name of the storage disk to use. + */ + protected function getStorageDiskName(): string + { + $storageType = config('filesystems.attachments'); + + // Change to our secure-attachment disk if any of the local options + // are used to prevent escaping that location. + if ($storageType === 'local' || $storageType === 'local_secure' || $storageType === 'local_secure_restricted') { + $storageType = 'local_secure_attachments'; + } + + return $storageType; + } + + /** + * Change the originally provided path to fit any disk-specific requirements. + * This also ensures the path is kept to the expected root folders. + */ + protected function adjustPathForStorageDisk(string $path): string + { + $path = (new WhitespacePathNormalizer())->normalizePath(str_replace('uploads/files/', '', $path)); + + if ($this->getStorageDiskName() === 'local_secure_attachments') { + return $path; + } + + return 'uploads/files/' . $path; + } +} From c6109c708735434fdb30333ff4c24b4a80b0b749 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 3 Nov 2024 14:13:05 +0000 Subject: [PATCH 21/41] ZIP Imports: Added listing, show view, delete, activity --- app/Activity/ActivityType.php | 4 ++ app/Exports/Controllers/ImportController.php | 50 ++++++++++++++++++- app/Exports/Import.php | 23 ++++++++- app/Exports/ImportRepo.php | 32 ++++++++++++ app/Http/Controller.php | 4 +- lang/en/activities.php | 8 +++ lang/en/entities.php | 6 +++ resources/views/exports/import-show.blade.php | 38 ++++++++++++++ resources/views/exports/import.blade.php | 13 +++++ .../views/exports/parts/import.blade.php | 19 +++++++ routes/web.php | 2 + 11 files changed, 193 insertions(+), 6 deletions(-) create mode 100644 resources/views/exports/import-show.blade.php create mode 100644 resources/views/exports/parts/import.blade.php diff --git a/app/Activity/ActivityType.php b/app/Activity/ActivityType.php index 09b2ae73c..5ec9b9cf0 100644 --- a/app/Activity/ActivityType.php +++ b/app/Activity/ActivityType.php @@ -67,6 +67,10 @@ class ActivityType const WEBHOOK_UPDATE = 'webhook_update'; const WEBHOOK_DELETE = 'webhook_delete'; + const IMPORT_CREATE = 'import_create'; + const IMPORT_RUN = 'import_run'; + const IMPORT_DELETE = 'import_delete'; + /** * Get all the possible values. */ diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 640b4c108..582fff975 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -1,7 +1,10 @@ middleware('can:content-import'); } + /** + * Show the view to start a new import, and also list out the existing + * in progress imports that are visible to the user. + */ public function start(Request $request) { - // TODO - Show existing imports for user (or for all users if admin-level user) + // TODO - Test visibility access for listed items + $imports = $this->imports->getVisibleImports(); + + $this->setPageTitle(trans('entities.import')); return view('exports.import', [ + 'imports' => $imports, 'zipErrors' => session()->pull('validation_errors') ?? [], ]); } + /** + * Upload, validate and store an import file. + */ public function upload(Request $request) { $this->validate($request, [ @@ -39,6 +53,38 @@ class ImportController extends Controller return redirect('/import'); } - return redirect("imports/{$import->id}"); + $this->logActivity(ActivityType::IMPORT_CREATE, $import); + + return redirect($import->getUrl()); + } + + /** + * Show a pending import, with a form to allow progressing + * with the import process. + */ + public function show(int $id) + { + // TODO - Test visibility access + $import = $this->imports->findVisible($id); + + $this->setPageTitle(trans('entities.import_continue')); + + return view('exports.import-show', [ + 'import' => $import, + ]); + } + + /** + * Delete an active pending import from the filesystem and database. + */ + public function delete(int $id) + { + // TODO - Test visibility access + $import = $this->imports->findVisible($id); + $this->imports->deleteImport($import); + + $this->logActivity(ActivityType::IMPORT_DELETE, $import); + + return redirect('/import'); } } diff --git a/app/Exports/Import.php b/app/Exports/Import.php index c3ac3d529..520d8ea6c 100644 --- a/app/Exports/Import.php +++ b/app/Exports/Import.php @@ -2,6 +2,7 @@ namespace BookStack\Exports; +use BookStack\Activity\Models\Loggable; use Carbon\Carbon; use Illuminate\Database\Eloquent\Factories\HasFactory; use Illuminate\Database\Eloquent\Model; @@ -17,7 +18,7 @@ use Illuminate\Database\Eloquent\Model; * @property Carbon $created_at * @property Carbon $updated_at */ -class Import extends Model +class Import extends Model implements Loggable { use HasFactory; @@ -38,4 +39,24 @@ class Import extends Model return self::TYPE_PAGE; } + + public function getSizeString(): string + { + $mb = round($this->size / 1000000, 2); + return "{$mb} MB"; + } + + /** + * Get the URL to view/continue this import. + */ + public function getUrl(string $path = ''): string + { + $path = ltrim($path, '/'); + return url("/import/{$this->id}" . ($path ? '/' . $path : '')); + } + + public function logDescriptor(): string + { + return "({$this->id}) {$this->name}"; + } } diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index c8157967b..d7e169ad1 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -6,6 +6,7 @@ use BookStack\Exceptions\ZipValidationException; use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; use BookStack\Uploads\FileStorage; +use Illuminate\Database\Eloquent\Collection; use Symfony\Component\HttpFoundation\File\UploadedFile; class ImportRepo @@ -15,6 +16,31 @@ class ImportRepo ) { } + /** + * @return Collection + */ + public function getVisibleImports(): Collection + { + $query = Import::query(); + + if (!userCan('settings-manage')) { + $query->where('created_by', user()->id); + } + + return $query->get(); + } + + public function findVisible(int $id): Import + { + $query = Import::query(); + + if (!userCan('settings-manage')) { + $query->where('created_by', user()->id); + } + + return $query->findOrFail($id); + } + public function storeFromUpload(UploadedFile $file): Import { $zipPath = $file->getRealPath(); @@ -45,4 +71,10 @@ class ImportRepo return $import; } + + public function deleteImport(Import $import): void + { + $this->storage->delete($import->path); + $import->delete(); + } } diff --git a/app/Http/Controller.php b/app/Http/Controller.php index 8facf5dab..090cf523a 100644 --- a/app/Http/Controller.php +++ b/app/Http/Controller.php @@ -152,10 +152,8 @@ abstract class Controller extends BaseController /** * Log an activity in the system. - * - * @param string|Loggable $detail */ - protected function logActivity(string $type, $detail = ''): void + protected function logActivity(string $type, string|Loggable $detail = ''): void { Activity::add($type, $detail); } diff --git a/lang/en/activities.php b/lang/en/activities.php index 092398ef0..7c3454d41 100644 --- a/lang/en/activities.php +++ b/lang/en/activities.php @@ -84,6 +84,14 @@ return [ 'webhook_delete' => 'deleted webhook', 'webhook_delete_notification' => 'Webhook successfully deleted', + // Imports + 'import_create' => 'created import', + 'import_create_notification' => 'Import successfully uploaded', + 'import_run' => 'updated import', + 'import_run_notification' => 'Content successfully imported', + 'import_delete' => 'deleted import', + 'import_delete_notification' => 'Import successfully deleted', + // Users 'user_create' => 'created user', 'user_create_notification' => 'User successfully created', diff --git a/lang/en/entities.php b/lang/en/entities.php index 106147335..e2d8e47c5 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -48,6 +48,12 @@ return [ 'import_desc' => 'Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to import then press "Validate Import" to proceed. After the file has been uploaded and validated you\'ll be able to configure & confirm the import in the next view.', 'import_zip_select' => 'Select ZIP file to upload', 'import_zip_validation_errors' => 'Errors were detected while validating the provided ZIP file:', + 'import_pending' => 'Pending Imports', + 'import_pending_none' => 'No imports have been started.', + 'import_continue' => 'Continue Import', + 'import_run' => 'Run Import', + 'import_delete_confirm' => 'Are you sure you want to delete this import?', + 'import_delete_desc' => 'This will delete the uploaded import ZIP file, and cannot be undone.', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php new file mode 100644 index 000000000..843a05246 --- /dev/null +++ b/resources/views/exports/import-show.blade.php @@ -0,0 +1,38 @@ +@extends('layouts.simple') + +@section('body') + +
    + +
    +

    {{ trans('entities.import_continue') }}

    + + {{ csrf_field() }} + + +
    + {{ trans('common.cancel') }} +
    + +
    +

    {{ trans('entities.import_delete_confirm') }}

    +

    {{ trans('entities.import_delete_desc') }}

    + +
    +
    + +
    +
    +
    + +
    + {{ method_field('DELETE') }} + {{ csrf_field() }} +
    + +@stop diff --git a/resources/views/exports/import.blade.php b/resources/views/exports/import.blade.php index c4d7c8818..be9de4c0e 100644 --- a/resources/views/exports/import.blade.php +++ b/resources/views/exports/import.blade.php @@ -38,6 +38,19 @@
    + +
    +

    {{ trans('entities.import_pending') }}

    + @if(count($imports) === 0) +

    {{ trans('entities.import_pending_none') }}

    + @else +
    + @foreach($imports as $import) + @include('exports.parts.import', ['import' => $import]) + @endforeach +
    + @endif +
    @stop diff --git a/resources/views/exports/parts/import.blade.php b/resources/views/exports/parts/import.blade.php new file mode 100644 index 000000000..5ff6600f2 --- /dev/null +++ b/resources/views/exports/parts/import.blade.php @@ -0,0 +1,19 @@ +@php + $type = $import->getType(); +@endphp +
    +
    + @icon($type) {{ $import->name }} +
    +
    + @if($type === 'book') +
    @icon('chapter') {{ $import->chapter_count }}
    + @endif + @if($type === 'book' || $type === 'chapter') +
    @icon('page') {{ $import->page_count }}
    + @endif +
    {{ $import->getSizeString() }}
    +
    @icon('time'){{ $import->created_at->diffForHumans() }}
    +
    +
    \ No newline at end of file diff --git a/routes/web.php b/routes/web.php index 91aab13fe..c490bb3b3 100644 --- a/routes/web.php +++ b/routes/web.php @@ -209,6 +209,8 @@ Route::middleware('auth')->group(function () { // Importing Route::get('/import', [ExportControllers\ImportController::class, 'start']); Route::post('/import', [ExportControllers\ImportController::class, 'upload']); + Route::get('/import/{id}', [ExportControllers\ImportController::class, 'show']); + Route::delete('/import/{id}', [ExportControllers\ImportController::class, 'delete']); // Other Pages Route::get('/', [HomeController::class, 'index']); From 8f6f81948e81b4d63251bee57da57aa5809eaad2 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 3 Nov 2024 17:28:18 +0000 Subject: [PATCH 22/41] ZIP Imports: Fleshed out continue page, Added testing --- app/Exports/Controllers/ImportController.php | 17 ++- app/Exports/Import.php | 9 ++ lang/en/entities.php | 4 + resources/views/exports/import-show.blade.php | 41 ++++- routes/web.php | 1 + tests/Exports/ZipImportTest.php | 140 ++++++++++++++++++ 6 files changed, 206 insertions(+), 6 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 582fff975..787fd1b27 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -23,9 +23,8 @@ class ImportController extends Controller * Show the view to start a new import, and also list out the existing * in progress imports that are visible to the user. */ - public function start(Request $request) + public function start() { - // TODO - Test visibility access for listed items $imports = $this->imports->getVisibleImports(); $this->setPageTitle(trans('entities.import')); @@ -64,7 +63,6 @@ class ImportController extends Controller */ public function show(int $id) { - // TODO - Test visibility access $import = $this->imports->findVisible($id); $this->setPageTitle(trans('entities.import_continue')); @@ -74,12 +72,23 @@ class ImportController extends Controller ]); } + public function run(int $id) + { + // TODO - Test access/visibility + + $import = $this->imports->findVisible($id); + + // TODO - Run import + // Validate again before + // TODO - Redirect to result + // TOOD - Or redirect back with errors + } + /** * Delete an active pending import from the filesystem and database. */ public function delete(int $id) { - // TODO - Test visibility access $import = $this->imports->findVisible($id); $this->imports->deleteImport($import); diff --git a/app/Exports/Import.php b/app/Exports/Import.php index 520d8ea6c..8400382fd 100644 --- a/app/Exports/Import.php +++ b/app/Exports/Import.php @@ -3,11 +3,14 @@ namespace BookStack\Exports; use BookStack\Activity\Models\Loggable; +use BookStack\Users\Models\User; use Carbon\Carbon; use Illuminate\Database\Eloquent\Factories\HasFactory; use Illuminate\Database\Eloquent\Model; +use Illuminate\Database\Eloquent\Relations\BelongsTo; /** + * @property int $id * @property string $path * @property string $name * @property int $size - ZIP size in bytes @@ -17,6 +20,7 @@ use Illuminate\Database\Eloquent\Model; * @property int $created_by * @property Carbon $created_at * @property Carbon $updated_at + * @property User $createdBy */ class Import extends Model implements Loggable { @@ -59,4 +63,9 @@ class Import extends Model implements Loggable { return "({$this->id}) {$this->name}"; } + + public function createdBy(): BelongsTo + { + return $this->belongsTo(User::class, 'created_by'); + } } diff --git a/lang/en/entities.php b/lang/en/entities.php index e2d8e47c5..4f5a53004 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -51,7 +51,11 @@ return [ 'import_pending' => 'Pending Imports', 'import_pending_none' => 'No imports have been started.', 'import_continue' => 'Continue Import', + 'import_continue_desc' => 'Review the content due to be imported from the uploaded ZIP file. When ready, run the import to add its contents to this system. The uploaded ZIP import file will be automatically removed on successful import.', 'import_run' => 'Run Import', + 'import_size' => 'Import ZIP Size:', + 'import_uploaded_at' => 'Uploaded:', + 'import_uploaded_by' => 'Uploaded by:', 'import_delete_confirm' => 'Are you sure you want to delete this import?', 'import_delete_desc' => 'This will delete the uploaded import ZIP file, and cannot be undone.', diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php index 843a05246..ac1b8a45d 100644 --- a/resources/views/exports/import-show.blade.php +++ b/resources/views/exports/import-show.blade.php @@ -6,7 +6,44 @@

    {{ trans('entities.import_continue') }}

    -
    +

    {{ trans('entities.import_continue_desc') }}

    + +
    + @php + $type = $import->getType(); + @endphp +
    +
    +

    @icon($type) {{ $import->name }}

    + @if($type === 'book') +

    @icon('chapter') {{ trans_choice('entities.x_chapters', $import->chapter_count) }}

    + @endif + @if($type === 'book' || $type === 'chapter') +

    @icon('page') {{ trans_choice('entities.x_pages', $import->page_count) }}

    + @endif +
    +
    +
    + {{ trans('entities.import_size') }} + {{ $import->getSizeString() }} +
    +
    + {{ trans('entities.import_uploaded_at') }} + {{ $import->created_at->diffForHumans() }} +
    + @if($import->createdBy) +
    + {{ trans('entities.import_uploaded_by') }} + {{ $import->createdBy->name }} +
    + @endif +
    +
    +
    + + {{ csrf_field() }}
    @@ -23,7 +60,7 @@
    - + diff --git a/routes/web.php b/routes/web.php index c490bb3b3..85f833528 100644 --- a/routes/web.php +++ b/routes/web.php @@ -210,6 +210,7 @@ Route::middleware('auth')->group(function () { Route::get('/import', [ExportControllers\ImportController::class, 'start']); Route::post('/import', [ExportControllers\ImportController::class, 'upload']); Route::get('/import/{id}', [ExportControllers\ImportController::class, 'show']); + Route::post('/import/{id}', [ExportControllers\ImportController::class, 'run']); Route::delete('/import/{id}', [ExportControllers\ImportController::class, 'delete']); // Other Pages diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php index c9d255b1e..b9a8598fa 100644 --- a/tests/Exports/ZipImportTest.php +++ b/tests/Exports/ZipImportTest.php @@ -2,6 +2,8 @@ namespace Tests\Exports; +use BookStack\Activity\ActivityType; +use BookStack\Exports\Import; use Illuminate\Http\UploadedFile; use Illuminate\Testing\TestResponse; use Tests\TestCase; @@ -35,6 +37,25 @@ class ZipImportTest extends TestCase $resp->assertSeeText('Select ZIP file to upload'); } + public function test_import_page_pending_import_visibility_limited() + { + $user = $this->users->viewer(); + $admin = $this->users->admin(); + $userImport = Import::factory()->create(['name' => 'MySuperUserImport', 'created_by' => $user->id]); + $adminImport = Import::factory()->create(['name' => 'MySuperAdminImport', 'created_by' => $admin->id]); + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $resp = $this->actingAs($user)->get('/import'); + $resp->assertSeeText('MySuperUserImport'); + $resp->assertDontSeeText('MySuperAdminImport'); + + $this->permissions->grantUserRolePermissions($user, ['settings-manage']); + + $resp = $this->actingAs($user)->get('/import'); + $resp->assertSeeText('MySuperUserImport'); + $resp->assertSeeText('MySuperAdminImport'); + } + public function test_zip_read_errors_are_shown_on_validation() { $invalidUpload = $this->files->uploadedImage('image.zip'); @@ -105,6 +126,125 @@ class ZipImportTest extends TestCase $resp->assertSeeText('[book.pages.1.tags.0.value]: The value must be a string.'); } + public function test_import_upload_success() + { + $admin = $this->users->admin(); + $this->actingAs($admin); + $resp = $this->runImportFromFile($this->zipUploadFromData([ + 'book' => [ + 'name' => 'My great book name', + 'chapters' => [ + [ + 'name' => 'my chapter', + 'pages' => [ + [ + 'name' => 'my chapter page', + ] + ] + ] + ], + 'pages' => [ + [ + 'name' => 'My page', + ] + ], + ], + ])); + + $this->assertDatabaseHas('imports', [ + 'name' => 'My great book name', + 'book_count' => 1, + 'chapter_count' => 1, + 'page_count' => 2, + 'created_by' => $admin->id, + ]); + + /** @var Import $import */ + $import = Import::query()->latest()->first(); + $resp->assertRedirect("/import/{$import->id}"); + $this->assertFileExists(storage_path($import->path)); + $this->assertActivityExists(ActivityType::IMPORT_CREATE); + } + + public function test_import_show_page() + { + $import = Import::factory()->create(['name' => 'MySuperAdminImport']); + + $resp = $this->asAdmin()->get("/import/{$import->id}"); + $resp->assertOk(); + $resp->assertSee('MySuperAdminImport'); + } + + public function test_import_show_page_access_limited() + { + $user = $this->users->viewer(); + $admin = $this->users->admin(); + $userImport = Import::factory()->create(['name' => 'MySuperUserImport', 'created_by' => $user->id]); + $adminImport = Import::factory()->create(['name' => 'MySuperAdminImport', 'created_by' => $admin->id]); + $this->actingAs($user); + + $this->get("/import/{$userImport->id}")->assertRedirect('/'); + $this->get("/import/{$adminImport->id}")->assertRedirect('/'); + + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $this->get("/import/{$userImport->id}")->assertOk(); + $this->get("/import/{$adminImport->id}")->assertStatus(404); + + $this->permissions->grantUserRolePermissions($user, ['settings-manage']); + + $this->get("/import/{$userImport->id}")->assertOk(); + $this->get("/import/{$adminImport->id}")->assertOk(); + } + + public function test_import_delete() + { + $this->asAdmin(); + $this->runImportFromFile($this->zipUploadFromData([ + 'book' => [ + 'name' => 'My great book name' + ], + ])); + + /** @var Import $import */ + $import = Import::query()->latest()->first(); + $this->assertDatabaseHas('imports', [ + 'id' => $import->id, + 'name' => 'My great book name' + ]); + $this->assertFileExists(storage_path($import->path)); + + $resp = $this->delete("/import/{$import->id}"); + + $resp->assertRedirect('/import'); + $this->assertActivityExists(ActivityType::IMPORT_DELETE); + $this->assertDatabaseMissing('imports', [ + 'id' => $import->id, + ]); + $this->assertFileDoesNotExist(storage_path($import->path)); + } + + public function test_import_delete_access_limited() + { + $user = $this->users->viewer(); + $admin = $this->users->admin(); + $userImport = Import::factory()->create(['name' => 'MySuperUserImport', 'created_by' => $user->id]); + $adminImport = Import::factory()->create(['name' => 'MySuperAdminImport', 'created_by' => $admin->id]); + $this->actingAs($user); + + $this->delete("/import/{$userImport->id}")->assertRedirect('/'); + $this->delete("/import/{$adminImport->id}")->assertRedirect('/'); + + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $this->delete("/import/{$userImport->id}")->assertRedirect('/import'); + $this->delete("/import/{$adminImport->id}")->assertStatus(404); + + $this->permissions->grantUserRolePermissions($user, ['settings-manage']); + + $this->delete("/import/{$adminImport->id}")->assertRedirect('/import'); + } + protected function runImportFromFile(UploadedFile $file): TestResponse { return $this->call('POST', '/import', [], [], ['file' => $file]); From 14578c22570d7f9ac197125ece1cf86d9d07be9b Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 4 Nov 2024 16:21:22 +0000 Subject: [PATCH 23/41] ZIP Imports: Added parent selector for page/chapter imports --- app/Exports/Controllers/ImportController.php | 14 ++++++++--- lang/en/entities.php | 2 ++ resources/sass/styles.scss | 18 +++++++++++++ resources/views/entities/selector.blade.php | 8 ++++++ resources/views/exports/import-show.blade.php | 25 +++++++++++++++---- resources/views/form/errors.blade.php | 3 +++ 6 files changed, 62 insertions(+), 8 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 787fd1b27..a2389c725 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -72,14 +72,22 @@ class ImportController extends Controller ]); } - public function run(int $id) + public function run(int $id, Request $request) { // TODO - Test access/visibility - $import = $this->imports->findVisible($id); + $parent = null; + + if ($import->getType() === 'page' || $import->getType() === 'chapter') { + $data = $this->validate($request, [ + 'parent' => ['required', 'string'] + ]); + $parent = $data['parent']; + } // TODO - Run import - // Validate again before + // TODO - Validate again before + // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) // TODO - Redirect to result // TOOD - Or redirect back with errors } diff --git a/lang/en/entities.php b/lang/en/entities.php index 4f5a53004..065eb043a 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -56,6 +56,8 @@ return [ 'import_size' => 'Import ZIP Size:', 'import_uploaded_at' => 'Uploaded:', 'import_uploaded_by' => 'Uploaded by:', + 'import_location' => 'Import Location', + 'import_location_desc' => 'Select a target location for your imported content. You\'ll need the relevant permissions to create within the location you choose.', 'import_delete_confirm' => 'Are you sure you want to delete this import?', 'import_delete_desc' => 'This will delete the uploaded import ZIP file, and cannot be undone.', diff --git a/resources/sass/styles.scss b/resources/sass/styles.scss index 636367e3a..942265d04 100644 --- a/resources/sass/styles.scss +++ b/resources/sass/styles.scss @@ -138,6 +138,11 @@ $loadingSize: 10px; font-size: 16px; padding: $-s $-m; } + input[type="text"]:focus { + outline: 1px solid var(--color-primary); + border-radius: 3px 3px 0 0; + outline-offset: -1px; + } .entity-list { overflow-y: scroll; height: 400px; @@ -171,6 +176,19 @@ $loadingSize: 10px; font-size: 14px; } } + &.small { + .entity-list-item { + padding: $-xs $-m; + } + .entity-list, .loading { + height: 300px; + } + input[type="text"] { + font-size: 13px; + padding: $-xs $-m; + height: auto; + } + } } .fullscreen { diff --git a/resources/views/entities/selector.blade.php b/resources/views/entities/selector.blade.php index c1280cfb2..0cdf4376c 100644 --- a/resources/views/entities/selector.blade.php +++ b/resources/views/entities/selector.blade.php @@ -1,3 +1,11 @@ +{{-- +$name - string +$autofocus - boolean, optional +$entityTypes - string, optional +$entityPermission - string, optional +$selectorEndpoint - string, optional +$selectorSize - string, optional (compact) +--}}
    getType(); + @endphp +
    @@ -9,11 +13,9 @@

    {{ trans('entities.import_continue_desc') }}

    - @php - $type = $import->getType(); - @endphp +
    -
    +

    @icon($type) {{ $import->name }}

    @if($type === 'book')

    @icon('chapter') {{ trans_choice('entities.x_chapters', $import->chapter_count) }}

    @@ -22,7 +24,7 @@

    @icon('page') {{ trans_choice('entities.x_pages', $import->page_count) }}

    @endif
    -
    +
    {{ trans('entities.import_size') }} {{ $import->getSizeString() }} @@ -45,6 +47,19 @@ action="{{ $import->getUrl() }}" method="POST"> {{ csrf_field() }} + + @if($type === 'page' || $type === 'chapter') +
    + +

    {{ trans('entities.import_location_desc') }}

    + @include('entities.selector', [ + 'name' => 'parent', + 'entityTypes' => $type === 'page' ? 'chapter,book' : 'book', + 'entityPermission' => "{$type}-create", + 'selectorSize' => 'compact small', + ]) + @include('form.errors', ['name' => 'parent']) + @endif
    diff --git a/resources/views/form/errors.blade.php b/resources/views/form/errors.blade.php index 03cd4be88..72d41ee56 100644 --- a/resources/views/form/errors.blade.php +++ b/resources/views/form/errors.blade.php @@ -1,3 +1,6 @@ +{{-- +$name - string +--}} @if($errors->has($name))
    {{ $errors->first($name) }}
    @endif \ No newline at end of file From 92cfde495e0d3141af608ea3734b612402f257dd Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 5 Nov 2024 13:17:31 +0000 Subject: [PATCH 24/41] ZIP Imports: Added full contents view to import display Reduced import data will now be stored on the import itself, instead of storing a set of totals. --- app/Exports/Controllers/ImportController.php | 5 ++- app/Exports/Import.php | 37 ++++++++---------- app/Exports/ImportRepo.php | 35 ++++++++++++++--- .../ZipExports/Models/ZipExportAttachment.php | 18 +++++++++ .../ZipExports/Models/ZipExportBook.php | 30 ++++++++++++++ .../ZipExports/Models/ZipExportChapter.php | 26 +++++++++++++ .../ZipExports/Models/ZipExportImage.php | 17 ++++++++ .../ZipExports/Models/ZipExportModel.php | 28 +++++++++++++ .../ZipExports/Models/ZipExportPage.php | 31 +++++++++++++++ .../ZipExports/Models/ZipExportTag.php | 16 ++++++++ app/Exports/ZipExports/ZipExportReader.php | 32 ++++++--------- app/Exports/ZipExports/ZipExportValidator.php | 1 - database/factories/Exports/ImportFactory.php | 5 +-- ...2024_11_02_160700_create_imports_table.php | 7 ++-- lang/en/entities.php | 8 ++-- resources/sass/styles.scss | 5 +++ resources/views/exports/import-show.blade.php | 39 ++++++------------- .../views/exports/parts/import-item.blade.php | 26 +++++++++++++ .../views/exports/parts/import.blade.php | 11 +----- tests/Exports/ZipImportTest.php | 31 +++++++++++---- 20 files changed, 303 insertions(+), 105 deletions(-) create mode 100644 resources/views/exports/parts/import-item.blade.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index a2389c725..3a56ed034 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -65,10 +65,13 @@ class ImportController extends Controller { $import = $this->imports->findVisible($id); +// dd($import->decodeMetadata()); + $this->setPageTitle(trans('entities.import_continue')); return view('exports.import-show', [ 'import' => $import, + 'data' => $import->decodeMetadata(), ]); } @@ -89,7 +92,7 @@ class ImportController extends Controller // TODO - Validate again before // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) // TODO - Redirect to result - // TOOD - Or redirect back with errors + // TODO - Or redirect back with errors } /** diff --git a/app/Exports/Import.php b/app/Exports/Import.php index 8400382fd..9c1771c46 100644 --- a/app/Exports/Import.php +++ b/app/Exports/Import.php @@ -3,6 +3,9 @@ namespace BookStack\Exports; use BookStack\Activity\Models\Loggable; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Users\Models\User; use Carbon\Carbon; use Illuminate\Database\Eloquent\Factories\HasFactory; @@ -14,9 +17,8 @@ use Illuminate\Database\Eloquent\Relations\BelongsTo; * @property string $path * @property string $name * @property int $size - ZIP size in bytes - * @property int $book_count - * @property int $chapter_count - * @property int $page_count + * @property string $type + * @property string $metadata * @property int $created_by * @property Carbon $created_at * @property Carbon $updated_at @@ -26,24 +28,6 @@ class Import extends Model implements Loggable { use HasFactory; - public const TYPE_BOOK = 'book'; - public const TYPE_CHAPTER = 'chapter'; - public const TYPE_PAGE = 'page'; - - /** - * Get the type (model) that this import is intended to be. - */ - public function getType(): string - { - if ($this->book_count === 1) { - return self::TYPE_BOOK; - } elseif ($this->chapter_count === 1) { - return self::TYPE_CHAPTER; - } - - return self::TYPE_PAGE; - } - public function getSizeString(): string { $mb = round($this->size / 1000000, 2); @@ -68,4 +52,15 @@ class Import extends Model implements Loggable { return $this->belongsTo(User::class, 'created_by'); } + + public function decodeMetadata(): ZipExportBook|ZipExportChapter|ZipExportPage|null + { + $metadataArray = json_decode($this->metadata, true); + return match ($this->type) { + 'book' => ZipExportBook::fromArray($metadataArray), + 'chapter' => ZipExportChapter::fromArray($metadataArray), + 'page' => ZipExportPage::fromArray($metadataArray), + default => null, + }; + } } diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index d7e169ad1..3265e1c80 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -2,7 +2,12 @@ namespace BookStack\Exports; +use BookStack\Exceptions\FileUploadException; +use BookStack\Exceptions\ZipExportException; use BookStack\Exceptions\ZipValidationException; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; use BookStack\Uploads\FileStorage; @@ -41,6 +46,11 @@ class ImportRepo return $query->findOrFail($id); } + /** + * @throws FileUploadException + * @throws ZipValidationException + * @throws ZipExportException + */ public function storeFromUpload(UploadedFile $file): Import { $zipPath = $file->getRealPath(); @@ -50,15 +60,23 @@ class ImportRepo throw new ZipValidationException($errors); } - $zipEntityInfo = (new ZipExportReader($zipPath))->getEntityInfo(); + $reader = new ZipExportReader($zipPath); + $exportModel = $reader->decodeDataToExportModel(); + $import = new Import(); - $import->name = $zipEntityInfo['name']; - $import->book_count = $zipEntityInfo['book_count']; - $import->chapter_count = $zipEntityInfo['chapter_count']; - $import->page_count = $zipEntityInfo['page_count']; + $import->type = match (get_class($exportModel)) { + ZipExportPage::class => 'page', + ZipExportChapter::class => 'chapter', + ZipExportBook::class => 'book', + }; + + $import->name = $exportModel->name; $import->created_by = user()->id; $import->size = filesize($zipPath); + $exportModel->metadataOnly(); + $import->metadata = json_encode($exportModel); + $path = $this->storage->uploadFile( $file, 'uploads/files/imports/', @@ -72,6 +90,13 @@ class ImportRepo return $import; } + public function runImport(Import $import, ?string $parent = null) + { + // TODO - Download import zip (if needed) + // TODO - Validate zip file again + // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) + } + public function deleteImport(Import $import): void { $this->storage->delete($import->path); diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index e586b91b0..1dbdc7333 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -14,6 +14,11 @@ class ZipExportAttachment extends ZipExportModel public ?string $link = null; public ?string $file = null; + public function metadataOnly(): void + { + $this->order = $this->link = $this->file = null; + } + public static function fromModel(Attachment $model, ZipExportFiles $files): self { $instance = new self(); @@ -49,4 +54,17 @@ class ZipExportAttachment extends ZipExportModel return $context->validateData($data, $rules); } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->order = isset($data['order']) ? intval($data['order']) : null; + $model->link = $data['link'] ?? null; + $model->file = $data['file'] ?? null; + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php index 7e1f2d810..0dc4e93d4 100644 --- a/app/Exports/ZipExports/Models/ZipExportBook.php +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -21,6 +21,21 @@ class ZipExportBook extends ZipExportModel /** @var ZipExportTag[] */ public array $tags = []; + public function metadataOnly(): void + { + $this->description_html = $this->cover = null; + + foreach ($this->chapters as $chapter) { + $chapter->metadataOnly(); + } + foreach ($this->pages as $page) { + $page->metadataOnly(); + } + foreach ($this->tags as $tag) { + $tag->metadataOnly(); + } + } + public static function fromModel(Book $model, ZipExportFiles $files): self { $instance = new self(); @@ -71,4 +86,19 @@ class ZipExportBook extends ZipExportModel return $errors; } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->description_html = $data['description_html'] ?? null; + $model->cover = $data['cover'] ?? null; + $model->tags = ZipExportTag::fromManyArray($data['tags'] ?? []); + $model->pages = ZipExportPage::fromManyArray($data['pages'] ?? []); + $model->chapters = ZipExportChapter::fromManyArray($data['chapters'] ?? []); + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php index 03df31b70..50440d61a 100644 --- a/app/Exports/ZipExports/Models/ZipExportChapter.php +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -18,6 +18,18 @@ class ZipExportChapter extends ZipExportModel /** @var ZipExportTag[] */ public array $tags = []; + public function metadataOnly(): void + { + $this->description_html = $this->priority = null; + + foreach ($this->pages as $page) { + $page->metadataOnly(); + } + foreach ($this->tags as $tag) { + $tag->metadataOnly(); + } + } + public static function fromModel(Chapter $model, ZipExportFiles $files): self { $instance = new self(); @@ -61,4 +73,18 @@ class ZipExportChapter extends ZipExportModel return $errors; } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->description_html = $data['description_html'] ?? null; + $model->priority = isset($data['priority']) ? intval($data['priority']) : null; + $model->tags = ZipExportTag::fromManyArray($data['tags'] ?? []); + $model->pages = ZipExportPage::fromManyArray($data['pages'] ?? []); + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportImage.php b/app/Exports/ZipExports/Models/ZipExportImage.php index 3388c66df..691eb918f 100644 --- a/app/Exports/ZipExports/Models/ZipExportImage.php +++ b/app/Exports/ZipExports/Models/ZipExportImage.php @@ -25,6 +25,11 @@ class ZipExportImage extends ZipExportModel return $instance; } + public function metadataOnly(): void + { + // + } + public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ @@ -36,4 +41,16 @@ class ZipExportImage extends ZipExportModel return $context->validateData($data, $rules); } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->file = $data['file']; + $model->type = $data['type']; + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportModel.php b/app/Exports/ZipExports/Models/ZipExportModel.php index 4d66f010f..d3a8c3567 100644 --- a/app/Exports/ZipExports/Models/ZipExportModel.php +++ b/app/Exports/ZipExports/Models/ZipExportModel.php @@ -26,4 +26,32 @@ abstract class ZipExportModel implements JsonSerializable * item in the array for its own validation messages. */ abstract public static function validate(ZipValidationHelper $context, array $data): array; + + /** + * Decode the array of data into this export model. + */ + abstract public static function fromArray(array $data): self; + + /** + * Decode an array of array data into an array of export models. + * @param array[] $data + * @return self[] + */ + public static function fromManyArray(array $data): array + { + $results = []; + foreach ($data as $item) { + $results[] = static::fromArray($item); + } + return $results; + } + + /** + * Remove additional content in this model to reduce it down + * to just essential id/name values for identification. + * + * The result of this may be something that does not pass validation, but is + * simple for the purpose of creating a contents. + */ + abstract public function metadataOnly(): void; } diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index 2c8b9a88a..3a876e7aa 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -21,6 +21,21 @@ class ZipExportPage extends ZipExportModel /** @var ZipExportTag[] */ public array $tags = []; + public function metadataOnly(): void + { + $this->html = $this->markdown = $this->priority = null; + + foreach ($this->attachments as $attachment) { + $attachment->metadataOnly(); + } + foreach ($this->images as $image) { + $image->metadataOnly(); + } + foreach ($this->tags as $tag) { + $tag->metadataOnly(); + } + } + public static function fromModel(Page $model, ZipExportFiles $files): self { $instance = new self(); @@ -70,4 +85,20 @@ class ZipExportPage extends ZipExportModel return $errors; } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->id = $data['id'] ?? null; + $model->name = $data['name']; + $model->html = $data['html'] ?? null; + $model->markdown = $data['markdown'] ?? null; + $model->priority = isset($data['priority']) ? intval($data['priority']) : null; + $model->attachments = ZipExportAttachment::fromManyArray($data['attachments'] ?? []); + $model->images = ZipExportImage::fromManyArray($data['images'] ?? []); + $model->tags = ZipExportTag::fromManyArray($data['tags'] ?? []); + + return $model; + } } diff --git a/app/Exports/ZipExports/Models/ZipExportTag.php b/app/Exports/ZipExports/Models/ZipExportTag.php index 99abb811c..b6c9e338a 100644 --- a/app/Exports/ZipExports/Models/ZipExportTag.php +++ b/app/Exports/ZipExports/Models/ZipExportTag.php @@ -11,6 +11,11 @@ class ZipExportTag extends ZipExportModel public ?string $value = null; public ?int $order = null; + public function metadataOnly(): void + { + $this->value = $this->order = null; + } + public static function fromModel(Tag $model): self { $instance = new self(); @@ -36,4 +41,15 @@ class ZipExportTag extends ZipExportModel return $context->validateData($data, $rules); } + + public static function fromArray(array $data): self + { + $model = new self(); + + $model->name = $data['name']; + $model->value = $data['value'] ?? null; + $model->order = isset($data['order']) ? intval($data['order']) : null; + + return $model; + } } diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php index 7187a1889..c3e47da04 100644 --- a/app/Exports/ZipExports/ZipExportReader.php +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -3,6 +3,10 @@ namespace BookStack\Exports\ZipExports; use BookStack\Exceptions\ZipExportException; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportModel; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use ZipArchive; class ZipExportReader @@ -71,32 +75,18 @@ class ZipExportReader /** * @throws ZipExportException - * @returns array{name: string, book_count: int, chapter_count: int, page_count: int} */ - public function getEntityInfo(): array + public function decodeDataToExportModel(): ZipExportBook|ZipExportChapter|ZipExportPage { $data = $this->readData(); - $info = ['name' => '', 'book_count' => 0, 'chapter_count' => 0, 'page_count' => 0]; - if (isset($data['book'])) { - $info['name'] = $data['book']['name'] ?? ''; - $info['book_count']++; - $chapters = $data['book']['chapters'] ?? []; - $pages = $data['book']['pages'] ?? []; - $info['chapter_count'] += count($chapters); - $info['page_count'] += count($pages); - foreach ($chapters as $chapter) { - $info['page_count'] += count($chapter['pages'] ?? []); - } - } elseif (isset($data['chapter'])) { - $info['name'] = $data['chapter']['name'] ?? ''; - $info['chapter_count']++; - $info['page_count'] += count($data['chapter']['pages'] ?? []); - } elseif (isset($data['page'])) { - $info['name'] = $data['page']['name'] ?? ''; - $info['page_count']++; + return ZipExportBook::fromArray($data['book']); + } else if (isset($data['chapter'])) { + return ZipExportChapter::fromArray($data['chapter']); + } else if (isset($data['page'])) { + return ZipExportPage::fromArray($data['page']); } - return $info; + throw new ZipExportException("Could not identify content in ZIP file data."); } } diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index e476998c2..e27ae53c7 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -38,7 +38,6 @@ class ZipExportValidator return ['format' => trans('errors.import_zip_no_data')]; } - return $this->flattenModelErrors($modelErrors, $keyPrefix); } diff --git a/database/factories/Exports/ImportFactory.php b/database/factories/Exports/ImportFactory.php index 55378d583..74a2bcd65 100644 --- a/database/factories/Exports/ImportFactory.php +++ b/database/factories/Exports/ImportFactory.php @@ -23,9 +23,8 @@ class ImportFactory extends Factory return [ 'path' => 'uploads/imports/' . Str::random(10) . '.zip', 'name' => $this->faker->words(3, true), - 'book_count' => 1, - 'chapter_count' => 5, - 'page_count' => 15, + 'type' => 'book', + 'metadata' => '{"name": "My book"}', 'created_at' => User::factory(), ]; } diff --git a/database/migrations/2024_11_02_160700_create_imports_table.php b/database/migrations/2024_11_02_160700_create_imports_table.php index ed1882269..0784591b8 100644 --- a/database/migrations/2024_11_02_160700_create_imports_table.php +++ b/database/migrations/2024_11_02_160700_create_imports_table.php @@ -16,10 +16,9 @@ return new class extends Migration $table->string('name'); $table->string('path'); $table->integer('size'); - $table->integer('book_count'); - $table->integer('chapter_count'); - $table->integer('page_count'); - $table->integer('created_by'); + $table->string('type'); + $table->longText('metadata'); + $table->integer('created_by')->index(); $table->timestamps(); }); } diff --git a/lang/en/entities.php b/lang/en/entities.php index 065eb043a..ae1c1e8d4 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -45,7 +45,7 @@ return [ 'default_template_select' => 'Select a template page', 'import' => 'Import', 'import_validate' => 'Validate Import', - 'import_desc' => 'Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to import then press "Validate Import" to proceed. After the file has been uploaded and validated you\'ll be able to configure & confirm the import in the next view.', + 'import_desc' => 'Import books, chapters & pages using a portable zip export from the same, or a different, instance. Select a ZIP file to proceed. After the file has been uploaded and validated you\'ll be able to configure & confirm the import in the next view.', 'import_zip_select' => 'Select ZIP file to upload', 'import_zip_validation_errors' => 'Errors were detected while validating the provided ZIP file:', 'import_pending' => 'Pending Imports', @@ -53,9 +53,9 @@ return [ 'import_continue' => 'Continue Import', 'import_continue_desc' => 'Review the content due to be imported from the uploaded ZIP file. When ready, run the import to add its contents to this system. The uploaded ZIP import file will be automatically removed on successful import.', 'import_run' => 'Run Import', - 'import_size' => 'Import ZIP Size:', - 'import_uploaded_at' => 'Uploaded:', - 'import_uploaded_by' => 'Uploaded by:', + 'import_size' => ':size Import ZIP Size', + 'import_uploaded_at' => 'Uploaded :relativeTime', + 'import_uploaded_by' => 'Uploaded by', 'import_location' => 'Import Location', 'import_location_desc' => 'Select a target location for your imported content. You\'ll need the relevant permissions to create within the location you choose.', 'import_delete_confirm' => 'Are you sure you want to delete this import?', diff --git a/resources/sass/styles.scss b/resources/sass/styles.scss index 942265d04..2cf3cbf82 100644 --- a/resources/sass/styles.scss +++ b/resources/sass/styles.scss @@ -248,4 +248,9 @@ $loadingSize: 10px; transform: rotate(180deg); } } +} + +.import-item { + border-inline-start: 2px solid currentColor; + padding-inline-start: $-xs; } \ No newline at end of file diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php index 63977947d..40867377f 100644 --- a/resources/views/exports/import-show.blade.php +++ b/resources/views/exports/import-show.blade.php @@ -1,11 +1,6 @@ @extends('layouts.simple') @section('body') - - @php - $type = $import->getType(); - @endphp -
    @@ -13,29 +8,17 @@

    {{ trans('entities.import_continue_desc') }}

    - -
    + +
    -

    @icon($type) {{ $import->name }}

    - @if($type === 'book') -

    @icon('chapter') {{ trans_choice('entities.x_chapters', $import->chapter_count) }}

    - @endif - @if($type === 'book' || $type === 'chapter') -

    @icon('page') {{ trans_choice('entities.x_pages', $import->page_count) }}

    - @endif + @include('exports.parts.import-item', ['type' => $import->type, 'model' => $data])
    -
    -
    - {{ trans('entities.import_size') }} - {{ $import->getSizeString() }} -
    -
    - {{ trans('entities.import_uploaded_at') }} - {{ $import->created_at->diffForHumans() }} -
    +
    +
    {{ trans('entities.import_size', ['size' => $import->getSizeString()]) }}
    +
    {{ trans('entities.import_uploaded_at', ['relativeTime' => $import->created_at->diffForHumans()]) }}
    @if($import->createdBy) -
    - {{ trans('entities.import_uploaded_by') }} +
    + {{ trans('entities.import_uploaded_by') }} {{ $import->createdBy->name }}
    @endif @@ -48,14 +31,14 @@ method="POST"> {{ csrf_field() }} - @if($type === 'page' || $type === 'chapter') + @if($import->type === 'page' || $import->type === 'chapter')

    {{ trans('entities.import_location_desc') }}

    @include('entities.selector', [ 'name' => 'parent', - 'entityTypes' => $type === 'page' ? 'chapter,book' : 'book', - 'entityPermission' => "{$type}-create", + 'entityTypes' => $import->type === 'page' ? 'chapter,book' : 'book', + 'entityPermission' => "{$import->type}-create", 'selectorSize' => 'compact small', ]) @include('form.errors', ['name' => 'parent']) diff --git a/resources/views/exports/parts/import-item.blade.php b/resources/views/exports/parts/import-item.blade.php new file mode 100644 index 000000000..811a3b31b --- /dev/null +++ b/resources/views/exports/parts/import-item.blade.php @@ -0,0 +1,26 @@ +{{-- +$type - string +$model - object +--}} +
    +

    @icon($type){{ $model->name }}

    +
    +
    + @if($model->attachments ?? []) + @icon('attach'){{ count($model->attachments) }} + @endif + @if($model->images ?? []) + @icon('image'){{ count($model->images) }} + @endif + @if($model->tags ?? []) + @icon('tag'){{ count($model->tags) }} + @endif +
    + @foreach($model->chapters ?? [] as $chapter) + @include('exports.parts.import-item', ['type' => 'chapter', 'model' => $chapter]) + @endforeach + @foreach($model->pages ?? [] as $page) + @include('exports.parts.import-item', ['type' => 'page', 'model' => $page]) + @endforeach +
    +
    \ No newline at end of file diff --git a/resources/views/exports/parts/import.blade.php b/resources/views/exports/parts/import.blade.php index 5ff6600f2..fd53095a4 100644 --- a/resources/views/exports/parts/import.blade.php +++ b/resources/views/exports/parts/import.blade.php @@ -1,18 +1,9 @@ -@php - $type = $import->getType(); -@endphp
    @icon($type) {{ $import->name }} + class="text-{{ $import->type }}">@icon($import->type) {{ $import->name }}
    - @if($type === 'book') -
    @icon('chapter') {{ $import->chapter_count }}
    - @endif - @if($type === 'book' || $type === 'chapter') -
    @icon('page') {{ $import->page_count }}
    - @endif
    {{ $import->getSizeString() }}
    @icon('time'){{ $import->created_at->diffForHumans() }}
    diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php index b9a8598fa..2b40100aa 100644 --- a/tests/Exports/ZipImportTest.php +++ b/tests/Exports/ZipImportTest.php @@ -4,6 +4,9 @@ namespace Tests\Exports; use BookStack\Activity\ActivityType; use BookStack\Exports\Import; +use BookStack\Exports\ZipExports\Models\ZipExportBook; +use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportPage; use Illuminate\Http\UploadedFile; use Illuminate\Testing\TestResponse; use Tests\TestCase; @@ -130,7 +133,7 @@ class ZipImportTest extends TestCase { $admin = $this->users->admin(); $this->actingAs($admin); - $resp = $this->runImportFromFile($this->zipUploadFromData([ + $data = [ 'book' => [ 'name' => 'My great book name', 'chapters' => [ @@ -149,13 +152,13 @@ class ZipImportTest extends TestCase ] ], ], - ])); + ]; + + $resp = $this->runImportFromFile($this->zipUploadFromData($data)); $this->assertDatabaseHas('imports', [ 'name' => 'My great book name', - 'book_count' => 1, - 'chapter_count' => 1, - 'page_count' => 2, + 'type' => 'book', 'created_by' => $admin->id, ]); @@ -168,11 +171,25 @@ class ZipImportTest extends TestCase public function test_import_show_page() { - $import = Import::factory()->create(['name' => 'MySuperAdminImport']); + $exportBook = new ZipExportBook(); + $exportBook->name = 'My exported book'; + $exportChapter = new ZipExportChapter(); + $exportChapter->name = 'My exported chapter'; + $exportPage = new ZipExportPage(); + $exportPage->name = 'My exported page'; + $exportBook->chapters = [$exportChapter]; + $exportChapter->pages = [$exportPage]; + + $import = Import::factory()->create([ + 'name' => 'MySuperAdminImport', + 'metadata' => json_encode($exportBook) + ]); $resp = $this->asAdmin()->get("/import/{$import->id}"); $resp->assertOk(); - $resp->assertSee('MySuperAdminImport'); + $resp->assertSeeText('My exported book'); + $resp->assertSeeText('My exported chapter'); + $resp->assertSeeText('My exported page'); } public function test_import_show_page_access_limited() From 7b84558ca1deb0a605a2f632e60baaad325615e7 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 5 Nov 2024 15:41:58 +0000 Subject: [PATCH 25/41] ZIP Imports: Added parent and permission check pre-import --- app/Exceptions/ZipImportException.php | 12 ++ app/Exports/Controllers/ImportController.php | 2 - app/Exports/ImportRepo.php | 20 ++- app/Exports/ZipExports/ZipExportValidator.php | 7 +- app/Exports/ZipExports/ZipImportRunner.php | 143 ++++++++++++++++++ app/Uploads/FileStorage.php | 23 ++- 6 files changed, 195 insertions(+), 12 deletions(-) create mode 100644 app/Exceptions/ZipImportException.php create mode 100644 app/Exports/ZipExports/ZipImportRunner.php diff --git a/app/Exceptions/ZipImportException.php b/app/Exceptions/ZipImportException.php new file mode 100644 index 000000000..2403c5144 --- /dev/null +++ b/app/Exceptions/ZipImportException.php @@ -0,0 +1,12 @@ +imports->findVisible($id); -// dd($import->decodeMetadata()); - $this->setPageTitle(trans('entities.import_continue')); return view('exports.import-show', [ diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index 3265e1c80..b94563545 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -2,6 +2,7 @@ namespace BookStack\Exports; +use BookStack\Entities\Queries\EntityQueries; use BookStack\Exceptions\FileUploadException; use BookStack\Exceptions\ZipExportException; use BookStack\Exceptions\ZipValidationException; @@ -10,6 +11,7 @@ use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; +use BookStack\Exports\ZipExports\ZipImportRunner; use BookStack\Uploads\FileStorage; use Illuminate\Database\Eloquent\Collection; use Symfony\Component\HttpFoundation\File\UploadedFile; @@ -18,6 +20,8 @@ class ImportRepo { public function __construct( protected FileStorage $storage, + protected ZipImportRunner $importer, + protected EntityQueries $entityQueries, ) { } @@ -54,13 +58,13 @@ class ImportRepo public function storeFromUpload(UploadedFile $file): Import { $zipPath = $file->getRealPath(); + $reader = new ZipExportReader($zipPath); - $errors = (new ZipExportValidator($zipPath))->validate(); + $errors = (new ZipExportValidator($reader))->validate(); if ($errors) { throw new ZipValidationException($errors); } - $reader = new ZipExportReader($zipPath); $exportModel = $reader->decodeDataToExportModel(); $import = new Import(); @@ -90,11 +94,17 @@ class ImportRepo return $import; } + /** + * @throws ZipValidationException + */ public function runImport(Import $import, ?string $parent = null) { - // TODO - Download import zip (if needed) - // TODO - Validate zip file again - // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) + $parentModel = null; + if ($import->type === 'page' || $import->type === 'chapter') { + $parentModel = $parent ? $this->entityQueries->findVisibleByStringIdentifier($parent) : null; + } + + return $this->importer->run($import, $parentModel); } public function deleteImport(Import $import): void diff --git a/app/Exports/ZipExports/ZipExportValidator.php b/app/Exports/ZipExports/ZipExportValidator.php index e27ae53c7..889804f20 100644 --- a/app/Exports/ZipExports/ZipExportValidator.php +++ b/app/Exports/ZipExports/ZipExportValidator.php @@ -10,20 +10,19 @@ use BookStack\Exports\ZipExports\Models\ZipExportPage; class ZipExportValidator { public function __construct( - protected string $zipPath, + protected ZipExportReader $reader, ) { } public function validate(): array { - $reader = new ZipExportReader($this->zipPath); try { - $importData = $reader->readData(); + $importData = $this->reader->readData(); } catch (ZipExportException $exception) { return ['format' => $exception->getMessage()]; } - $helper = new ZipValidationHelper($reader); + $helper = new ZipValidationHelper($this->reader); if (isset($importData['book'])) { $modelErrors = ZipExportBook::validate($helper, $importData['book']); diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php new file mode 100644 index 000000000..2f784ebea --- /dev/null +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -0,0 +1,143 @@ +getZipPath($import); + $reader = new ZipExportReader($zipPath); + + $errors = (new ZipExportValidator($reader))->validate(); + if ($errors) { + throw new ZipImportException(["ZIP failed to validate"]); + } + + try { + $exportModel = $reader->decodeDataToExportModel(); + } catch (ZipExportException $e) { + throw new ZipImportException([$e->getMessage()]); + } + + // Validate parent type + if ($exportModel instanceof ZipExportBook && ($parent !== null)) { + throw new ZipImportException(["Must not have a parent set for a Book import"]); + } else if ($exportModel instanceof ZipExportChapter && (!$parent instanceof Book)) { + throw new ZipImportException(["Parent book required for chapter import"]); + } else if ($exportModel instanceof ZipExportPage && !($parent instanceof Book || $parent instanceof Chapter)) { + throw new ZipImportException(["Parent book or chapter required for page import"]); + } + + $this->ensurePermissionsPermitImport($exportModel); + + // TODO - Run import + } + + /** + * @throws ZipImportException + */ + protected function ensurePermissionsPermitImport(ZipExportPage|ZipExportChapter|ZipExportBook $exportModel, Book|Chapter|null $parent = null): void + { + $errors = []; + + // TODO - Extract messages to language files + // TODO - Ensure these are shown to users on failure + + $chapters = []; + $pages = []; + $images = []; + $attachments = []; + + if ($exportModel instanceof ZipExportBook) { + if (!userCan('book-create-all')) { + $errors[] = 'You are lacking the required permission to create books.'; + } + array_push($pages, ...$exportModel->pages); + array_push($chapters, ...$exportModel->chapters); + } else if ($exportModel instanceof ZipExportChapter) { + $chapters[] = $exportModel; + } else if ($exportModel instanceof ZipExportPage) { + $pages[] = $exportModel; + } + + foreach ($chapters as $chapter) { + array_push($pages, ...$chapter->pages); + } + + if (count($chapters) > 0) { + $permission = 'chapter-create' . ($parent ? '' : '-all'); + if (!userCan($permission, $parent)) { + $errors[] = 'You are lacking the required permission to create chapters.'; + } + } + + foreach ($pages as $page) { + array_push($attachments, ...$page->attachments); + array_push($images, ...$page->images); + } + + if (count($pages) > 0) { + if ($parent) { + if (!userCan('page-create', $parent)) { + $errors[] = 'You are lacking the required permission to create pages.'; + } + } else { + $hasPermission = userCan('page-create-all') || userCan('page-create-own'); + if (!$hasPermission) { + $errors[] = 'You are lacking the required permission to create pages.'; + } + } + } + + if (count($images) > 0) { + if (!userCan('image-create-all')) { + $errors[] = 'You are lacking the required permissions to create images.'; + } + } + + if (count($attachments) > 0) { + if (userCan('attachment-create-all')) { + $errors[] = 'You are lacking the required permissions to create attachments.'; + } + } + + if (count($errors)) { + throw new ZipImportException($errors); + } + } + + protected function getZipPath(Import $import): string + { + if (!$this->storage->isRemote()) { + return $this->storage->getSystemPath($import->path); + } + + $tempFilePath = tempnam(sys_get_temp_dir(), 'bszip-import-'); + $tempFile = fopen($tempFilePath, 'wb'); + $stream = $this->storage->getReadStream($import->path); + stream_copy_to_stream($stream, $tempFile); + fclose($tempFile); + + return $tempFilePath; + } +} diff --git a/app/Uploads/FileStorage.php b/app/Uploads/FileStorage.php index 278484e51..e6ac368d0 100644 --- a/app/Uploads/FileStorage.php +++ b/app/Uploads/FileStorage.php @@ -5,6 +5,7 @@ namespace BookStack\Uploads; use BookStack\Exceptions\FileUploadException; use Exception; use Illuminate\Contracts\Filesystem\Filesystem as Storage; +use Illuminate\Filesystem\FilesystemAdapter; use Illuminate\Filesystem\FilesystemManager; use Illuminate\Support\Facades\Log; use Illuminate\Support\Str; @@ -70,6 +71,26 @@ class FileStorage return $filePath; } + /** + * Check whether the configured storage is remote from the host of this app. + */ + public function isRemote(): bool + { + return $this->getStorageDiskName() === 's3'; + } + + /** + * Get the actual path on system for the given relative file path. + */ + public function getSystemPath(string $filePath): string + { + if ($this->isRemote()) { + return ''; + } + + return storage_path('uploads/files/' . ltrim($this->adjustPathForStorageDisk($filePath), '/')); + } + /** * Get the storage that will be used for storing files. */ @@ -83,7 +104,7 @@ class FileStorage */ protected function getStorageDiskName(): string { - $storageType = config('filesystems.attachments'); + $storageType = trim(strtolower(config('filesystems.attachments'))); // Change to our secure-attachment disk if any of the local options // are used to prevent escaping that location. From d13e4d2eefeed427c0377be04761a639e9fdb8fc Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 9 Nov 2024 14:01:24 +0000 Subject: [PATCH 26/41] ZIP imports: Started actual import logic --- app/Entities/Tools/Cloner.php | 17 +-- .../ZipExports/Models/ZipExportAttachment.php | 6 +- .../ZipExports/Models/ZipExportTag.php | 6 +- app/Exports/ZipExports/ZipExportReader.php | 8 ++ app/Exports/ZipExports/ZipImportRunner.php | 107 ++++++++++++++++++ dev/docs/portable-zip-file-format.md | 4 +- 6 files changed, 124 insertions(+), 24 deletions(-) diff --git a/app/Entities/Tools/Cloner.php b/app/Entities/Tools/Cloner.php index 2030b050c..2be6083e3 100644 --- a/app/Entities/Tools/Cloner.php +++ b/app/Entities/Tools/Cloner.php @@ -18,17 +18,12 @@ use Illuminate\Http\UploadedFile; class Cloner { - protected PageRepo $pageRepo; - protected ChapterRepo $chapterRepo; - protected BookRepo $bookRepo; - protected ImageService $imageService; - - public function __construct(PageRepo $pageRepo, ChapterRepo $chapterRepo, BookRepo $bookRepo, ImageService $imageService) - { - $this->pageRepo = $pageRepo; - $this->chapterRepo = $chapterRepo; - $this->bookRepo = $bookRepo; - $this->imageService = $imageService; + public function __construct( + protected PageRepo $pageRepo, + protected ChapterRepo $chapterRepo, + protected BookRepo $bookRepo, + protected ImageService $imageService, + ) { } /** diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index 1dbdc7333..c6615e1dc 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -10,13 +10,12 @@ class ZipExportAttachment extends ZipExportModel { public ?int $id = null; public string $name; - public ?int $order = null; public ?string $link = null; public ?string $file = null; public function metadataOnly(): void { - $this->order = $this->link = $this->file = null; + $this->link = $this->file = null; } public static function fromModel(Attachment $model, ZipExportFiles $files): self @@ -24,7 +23,6 @@ class ZipExportAttachment extends ZipExportModel $instance = new self(); $instance->id = $model->id; $instance->name = $model->name; - $instance->order = $model->order; if ($model->external) { $instance->link = $model->path; @@ -47,7 +45,6 @@ class ZipExportAttachment extends ZipExportModel $rules = [ 'id' => ['nullable', 'int'], 'name' => ['required', 'string', 'min:1'], - 'order' => ['nullable', 'integer'], 'link' => ['required_without:file', 'nullable', 'string'], 'file' => ['required_without:link', 'nullable', 'string', $context->fileReferenceRule()], ]; @@ -61,7 +58,6 @@ class ZipExportAttachment extends ZipExportModel $model->id = $data['id'] ?? null; $model->name = $data['name']; - $model->order = isset($data['order']) ? intval($data['order']) : null; $model->link = $data['link'] ?? null; $model->file = $data['file'] ?? null; diff --git a/app/Exports/ZipExports/Models/ZipExportTag.php b/app/Exports/ZipExports/Models/ZipExportTag.php index b6c9e338a..6b4720fca 100644 --- a/app/Exports/ZipExports/Models/ZipExportTag.php +++ b/app/Exports/ZipExports/Models/ZipExportTag.php @@ -9,11 +9,10 @@ class ZipExportTag extends ZipExportModel { public string $name; public ?string $value = null; - public ?int $order = null; public function metadataOnly(): void { - $this->value = $this->order = null; + $this->value = null; } public static function fromModel(Tag $model): self @@ -21,7 +20,6 @@ class ZipExportTag extends ZipExportModel $instance = new self(); $instance->name = $model->name; $instance->value = $model->value; - $instance->order = $model->order; return $instance; } @@ -36,7 +34,6 @@ class ZipExportTag extends ZipExportModel $rules = [ 'name' => ['required', 'string', 'min:1'], 'value' => ['nullable', 'string'], - 'order' => ['nullable', 'integer'], ]; return $context->validateData($data, $rules); @@ -48,7 +45,6 @@ class ZipExportTag extends ZipExportModel $model->name = $data['name']; $model->value = $data['value'] ?? null; - $model->order = isset($data['order']) ? intval($data['order']) : null; return $model; } diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php index c3e47da04..ebc2fbbc9 100644 --- a/app/Exports/ZipExports/ZipExportReader.php +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -73,6 +73,14 @@ class ZipExportReader return $this->zip->statName("files/{$fileName}") !== false; } + /** + * @return false|resource + */ + public function streamFile(string $fileName) + { + return $this->zip->getStream("files/{$fileName}"); + } + /** * @throws ZipExportException */ diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 2f784ebea..2b897ff91 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -5,18 +5,33 @@ namespace BookStack\Exports\ZipExports; use BookStack\Entities\Models\Book; use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Entity; +use BookStack\Entities\Models\Page; +use BookStack\Entities\Repos\BookRepo; +use BookStack\Entities\Repos\ChapterRepo; +use BookStack\Entities\Repos\PageRepo; use BookStack\Exceptions\ZipExportException; use BookStack\Exceptions\ZipImportException; use BookStack\Exports\Import; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportPage; +use BookStack\Exports\ZipExports\Models\ZipExportTag; use BookStack\Uploads\FileStorage; +use BookStack\Uploads\ImageService; +use Illuminate\Http\UploadedFile; class ZipImportRunner { + protected array $tempFilesToCleanup = []; // TODO + protected array $createdImages = []; // TODO + protected array $createdAttachments = []; // TODO + public function __construct( protected FileStorage $storage, + protected PageRepo $pageRepo, + protected ChapterRepo $chapterRepo, + protected BookRepo $bookRepo, + protected ImageService $imageService, ) { } @@ -51,6 +66,98 @@ class ZipImportRunner $this->ensurePermissionsPermitImport($exportModel); // TODO - Run import + // TODO - In transaction? + // TODO - Revert uploaded files if goes wrong + } + + protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader): Book + { + $book = $this->bookRepo->create([ + 'name' => $exportBook->name, + 'description_html' => $exportBook->description_html ?? '', + 'image' => $exportBook->cover ? $this->zipFileToUploadedFile($exportBook->cover, $reader) : null, + 'tags' => $this->exportTagsToInputArray($exportBook->tags ?? []), + ]); + + // TODO - Parse/format description_html references + + if ($book->cover) { + $this->createdImages[] = $book->cover; + } + + // TODO - Pages + foreach ($exportBook->chapters as $exportChapter) { + $this->importChapter($exportChapter, $book); + } + // TODO - Sort chapters/pages by order + + return $book; + } + + protected function importChapter(ZipExportChapter $exportChapter, Book $parent, ZipExportReader $reader): Chapter + { + $chapter = $this->chapterRepo->create([ + 'name' => $exportChapter->name, + 'description_html' => $exportChapter->description_html ?? '', + 'tags' => $this->exportTagsToInputArray($exportChapter->tags ?? []), + ], $parent); + + // TODO - Parse/format description_html references + + $exportPages = $exportChapter->pages; + usort($exportPages, function (ZipExportPage $a, ZipExportPage $b) { + return ($a->priority ?? 0) - ($b->priority ?? 0); + }); + + foreach ($exportPages as $exportPage) { + // + } + // TODO - Pages + + return $chapter; + } + + protected function importPage(ZipExportPage $exportPage, Book|Chapter $parent, ZipExportReader $reader): Page + { + $page = $this->pageRepo->getNewDraftPage($parent); + + // TODO - Import attachments + // TODO - Import images + // TODO - Parse/format HTML + + $this->pageRepo->publishDraft($page, [ + 'name' => $exportPage->name, + 'markdown' => $exportPage->markdown, + 'html' => $exportPage->html, + 'tags' => $this->exportTagsToInputArray($exportPage->tags ?? []), + ]); + + return $page; + } + + protected function exportTagsToInputArray(array $exportTags): array + { + $tags = []; + + /** @var ZipExportTag $tag */ + foreach ($exportTags as $tag) { + $tags[] = ['name' => $tag->name, 'value' => $tag->value ?? '']; + } + + return $tags; + } + + protected function zipFileToUploadedFile(string $fileName, ZipExportReader $reader): UploadedFile + { + $tempPath = tempnam(sys_get_temp_dir(), 'bszipextract'); + $fileStream = $reader->streamFile($fileName); + $tempStream = fopen($tempPath, 'wb'); + stream_copy_to_stream($fileStream, $tempStream); + fclose($tempStream); + + $this->tempFilesToCleanup[] = $tempPath; + + return new UploadedFile($tempPath, $fileName); } /** diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 6cee7356d..7e5df3f01 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -135,12 +135,10 @@ embedded within it. - `name` - String, required, name of attachment. - `link` - String, semi-optional, URL of attachment. - `file` - String reference, semi-optional, reference to attachment file. -- `order` - Number, optional, integer order of the attachments (shown low to high). Either `link` or `file` must be present, as that will determine the type of attachment. #### Tag - `name` - String, required, name of the tag. -- `value` - String, optional, value of the tag (can be empty). -- `order` - Number, optional, integer order of the tags (shown low to high). \ No newline at end of file +- `value` - String, optional, value of the tag (can be empty). \ No newline at end of file From 378f0d595fe8aa5aca212e1c5ed22944bf8bf1b7 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sun, 10 Nov 2024 16:03:50 +0000 Subject: [PATCH 27/41] ZIP Imports: Built out reference parsing/updating logic --- app/Entities/Repos/PageRepo.php | 13 +- .../ZipExports/ZipExportReferences.php | 8 +- .../ZipExports/ZipImportReferences.php | 142 ++++++++++++++++++ app/Exports/ZipExports/ZipImportRunner.php | 20 ++- app/Exports/ZipExports/ZipReferenceParser.php | 72 +++++++-- 5 files changed, 232 insertions(+), 23 deletions(-) create mode 100644 app/Exports/ZipExports/ZipImportReferences.php diff --git a/app/Entities/Repos/PageRepo.php b/app/Entities/Repos/PageRepo.php index 1bc15392c..68b1c398f 100644 --- a/app/Entities/Repos/PageRepo.php +++ b/app/Entities/Repos/PageRepo.php @@ -87,6 +87,17 @@ class PageRepo return $draft; } + /** + * Directly update the content for the given page from the provided input. + * Used for direct content access in a way that performs required changes + * (Search index & reference regen) without performing an official update. + */ + public function setContentFromInput(Page $page, array $input): void + { + $this->updateTemplateStatusAndContentFromInput($page, $input); + $this->baseRepo->update($page, []); + } + /** * Update a page in the system. */ @@ -121,7 +132,7 @@ class PageRepo return $page; } - protected function updateTemplateStatusAndContentFromInput(Page $page, array $input) + protected function updateTemplateStatusAndContentFromInput(Page $page, array $input): void { if (isset($input['template']) && userCan('templates-manage')) { $page->template = ($input['template'] === 'true'); diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index c630c832b..0de409fa1 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -85,9 +85,9 @@ class ZipExportReferences // Parse page content first foreach ($this->pages as $page) { $handler = $createHandler($page); - $page->html = $this->parser->parse($page->html ?? '', $handler); + $page->html = $this->parser->parseLinks($page->html ?? '', $handler); if ($page->markdown) { - $page->markdown = $this->parser->parse($page->markdown, $handler); + $page->markdown = $this->parser->parseLinks($page->markdown, $handler); } } @@ -95,7 +95,7 @@ class ZipExportReferences foreach ($this->chapters as $chapter) { if ($chapter->description_html) { $handler = $createHandler($chapter); - $chapter->description_html = $this->parser->parse($chapter->description_html, $handler); + $chapter->description_html = $this->parser->parseLinks($chapter->description_html, $handler); } } @@ -103,7 +103,7 @@ class ZipExportReferences foreach ($this->books as $book) { if ($book->description_html) { $handler = $createHandler($book); - $book->description_html = $this->parser->parse($book->description_html, $handler); + $book->description_html = $this->parser->parseLinks($book->description_html, $handler); } } } diff --git a/app/Exports/ZipExports/ZipImportReferences.php b/app/Exports/ZipExports/ZipImportReferences.php new file mode 100644 index 000000000..8062886e5 --- /dev/null +++ b/app/Exports/ZipExports/ZipImportReferences.php @@ -0,0 +1,142 @@ + */ + protected array $referenceMap = []; + + /** @var array */ + protected array $zipExportPageMap = []; + /** @var array */ + protected array $zipExportChapterMap = []; + /** @var array */ + protected array $zipExportBookMap = []; + + public function __construct( + protected ZipReferenceParser $parser, + protected BaseRepo $baseRepo, + protected PageRepo $pageRepo, + protected ImageResizer $imageResizer, + ) { + } + + protected function addReference(string $type, Model $model, ?int $importId): void + { + if ($importId) { + $key = $type . ':' . $importId; + $this->referenceMap[$key] = $model; + } + } + + public function addPage(Page $page, ZipExportPage $exportPage): void + { + $this->pages[] = $page; + $this->zipExportPageMap[$page->id] = $exportPage; + $this->addReference('page', $page, $exportPage->id); + } + + public function addChapter(Chapter $chapter, ZipExportChapter $exportChapter): void + { + $this->chapters[] = $chapter; + $this->zipExportChapterMap[$chapter->id] = $exportChapter; + $this->addReference('chapter', $chapter, $exportChapter->id); + } + + public function addBook(Book $book, ZipExportBook $exportBook): void + { + $this->books[] = $book; + $this->zipExportBookMap[$book->id] = $exportBook; + $this->addReference('book', $book, $exportBook->id); + } + + public function addAttachment(Attachment $attachment, ?int $importId): void + { + $this->attachments[] = $attachment; + $this->addReference('attachment', $attachment, $importId); + } + + public function addImage(Image $image, ?int $importId): void + { + $this->images[] = $image; + $this->addReference('image', $image, $importId); + } + + protected function handleReference(string $type, int $id): ?string + { + $key = $type . ':' . $id; + $model = $this->referenceMap[$key] ?? null; + if ($model instanceof Entity) { + return $model->getUrl(); + } else if ($model instanceof Image) { + if ($model->type === 'gallery') { + $this->imageResizer->loadGalleryThumbnailsForImage($model, false); + return $model->thumbs['gallery'] ?? $model->url; + } + + return $model->url; + } + + return null; + } + + public function replaceReferences(): void + { + foreach ($this->books as $book) { + $exportBook = $this->zipExportBookMap[$book->id]; + $content = $exportBook->description_html || ''; + $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); + + $this->baseRepo->update($book, [ + 'description_html' => $parsed, + ]); + } + + foreach ($this->chapters as $chapter) { + $exportChapter = $this->zipExportChapterMap[$chapter->id]; + $content = $exportChapter->description_html || ''; + $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); + + $this->baseRepo->update($chapter, [ + 'description_html' => $parsed, + ]); + } + + foreach ($this->pages as $page) { + $exportPage = $this->zipExportPageMap[$page->id]; + $contentType = $exportPage->markdown ? 'markdown' : 'html'; + $content = $exportPage->markdown ?: ($exportPage->html ?: ''); + $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); + + $this->pageRepo->setContentFromInput($page, [ + $contentType => $parsed, + ]); + } + } +} diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 2b897ff91..345c22be1 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -23,8 +23,6 @@ use Illuminate\Http\UploadedFile; class ZipImportRunner { protected array $tempFilesToCleanup = []; // TODO - protected array $createdImages = []; // TODO - protected array $createdAttachments = []; // TODO public function __construct( protected FileStorage $storage, @@ -32,6 +30,7 @@ class ZipImportRunner protected ChapterRepo $chapterRepo, protected BookRepo $bookRepo, protected ImageService $imageService, + protected ZipImportReferences $references, ) { } @@ -68,6 +67,11 @@ class ZipImportRunner // TODO - Run import // TODO - In transaction? // TODO - Revert uploaded files if goes wrong + // TODO - Attachments + // TODO - Images + // (Both listed/stored in references) + + $this->references->replaceReferences(); } protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader): Book @@ -82,15 +86,17 @@ class ZipImportRunner // TODO - Parse/format description_html references if ($book->cover) { - $this->createdImages[] = $book->cover; + $this->references->addImage($book->cover, null); } // TODO - Pages foreach ($exportBook->chapters as $exportChapter) { - $this->importChapter($exportChapter, $book); + $this->importChapter($exportChapter, $book, $reader); } // TODO - Sort chapters/pages by order + $this->references->addBook($book, $exportBook); + return $book; } @@ -114,6 +120,8 @@ class ZipImportRunner } // TODO - Pages + $this->references->addChapter($chapter, $exportChapter); + return $chapter; } @@ -122,7 +130,9 @@ class ZipImportRunner $page = $this->pageRepo->getNewDraftPage($parent); // TODO - Import attachments + // TODO - Add attachment references // TODO - Import images + // TODO - Add image references // TODO - Parse/format HTML $this->pageRepo->publishDraft($page, [ @@ -132,6 +142,8 @@ class ZipImportRunner 'tags' => $this->exportTagsToInputArray($exportPage->tags ?? []), ]); + $this->references->addPage($page, $exportPage); + return $page; } diff --git a/app/Exports/ZipExports/ZipReferenceParser.php b/app/Exports/ZipExports/ZipReferenceParser.php index da43d1b36..5929383b4 100644 --- a/app/Exports/ZipExports/ZipReferenceParser.php +++ b/app/Exports/ZipExports/ZipReferenceParser.php @@ -15,27 +15,23 @@ use BookStack\References\ModelResolvers\PagePermalinkModelResolver; class ZipReferenceParser { /** - * @var CrossLinkModelResolver[] + * @var CrossLinkModelResolver[]|null */ - protected array $modelResolvers; + protected ?array $modelResolvers = null; - public function __construct(EntityQueries $queries) - { - $this->modelResolvers = [ - new PagePermalinkModelResolver($queries->pages), - new PageLinkModelResolver($queries->pages), - new ChapterLinkModelResolver($queries->chapters), - new BookLinkModelResolver($queries->books), - new ImageModelResolver(), - new AttachmentModelResolver(), - ]; + public function __construct( + protected EntityQueries $queries + ) { } /** * Parse and replace references in the given content. + * Calls the handler for each model link detected and replaces the link + * with the handler return value if provided. + * Returns the resulting content with links replaced. * @param callable(Model):(string|null) $handler */ - public function parse(string $content, callable $handler): string + public function parseLinks(string $content, callable $handler): string { $escapedBase = preg_quote(url('/'), '/'); $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#()]/"; @@ -59,13 +55,43 @@ class ZipReferenceParser return $content; } + /** + * Parse and replace references in the given content. + * Calls the handler for each reference detected and replaces the link + * with the handler return value if provided. + * Returns the resulting content string with references replaced. + * @param callable(string $type, int $id):(string|null) $handler + */ + public function parseReferences(string $content, callable $handler): string + { + $referenceRegex = '/\[\[bsexport:([a-z]+):(\d+)]]/'; + $matches = []; + preg_match_all($referenceRegex, $content, $matches); + + if (count($matches) < 3) { + return $content; + } + + for ($i = 0; $i < count($matches[0]); $i++) { + $referenceText = $matches[0][$i]; + $type = strtolower($matches[1][$i]); + $id = intval($matches[2][$i]); + $result = $handler($type, $id); + if ($result !== null) { + $content = str_replace($referenceText, $result, $content); + } + } + + return $content; + } + /** * Attempt to resolve the given link to a model using the instance model resolvers. */ protected function linkToModel(string $link): ?Model { - foreach ($this->modelResolvers as $resolver) { + foreach ($this->getModelResolvers() as $resolver) { $model = $resolver->resolve($link); if (!is_null($model)) { return $model; @@ -74,4 +100,22 @@ class ZipReferenceParser return null; } + + protected function getModelResolvers(): array + { + if (isset($this->modelResolvers)) { + return $this->modelResolvers; + } + + $this->modelResolvers = [ + new PagePermalinkModelResolver($this->queries->pages), + new PageLinkModelResolver($this->queries->pages), + new ChapterLinkModelResolver($this->queries->chapters), + new BookLinkModelResolver($this->queries->books), + new ImageModelResolver(), + new AttachmentModelResolver(), + ]; + + return $this->modelResolvers; + } } From 48c101aa7ab5b77781f4cd536b654d037b5aa55e Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 11 Nov 2024 15:06:46 +0000 Subject: [PATCH 28/41] ZIP Imports: Finished off core import logic --- app/Exceptions/ZipImportException.php | 3 +- app/Exports/Controllers/ImportController.php | 11 +- app/Exports/ImportRepo.php | 6 +- .../ZipExports/ZipImportReferences.php | 4 +- app/Exports/ZipExports/ZipImportRunner.php | 117 +++++++++++++++--- 5 files changed, 113 insertions(+), 28 deletions(-) diff --git a/app/Exceptions/ZipImportException.php b/app/Exceptions/ZipImportException.php index 2403c5144..452365c6e 100644 --- a/app/Exceptions/ZipImportException.php +++ b/app/Exceptions/ZipImportException.php @@ -7,6 +7,7 @@ class ZipImportException extends \Exception public function __construct( public array $errors ) { - parent::__construct(); + $message = "Import failed with errors:" . implode("\n", $this->errors); + parent::__construct($message); } } diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index ec5ac8080..4d2c83090 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -79,18 +79,21 @@ class ImportController extends Controller $import = $this->imports->findVisible($id); $parent = null; - if ($import->getType() === 'page' || $import->getType() === 'chapter') { + if ($import->type === 'page' || $import->type === 'chapter') { $data = $this->validate($request, [ 'parent' => ['required', 'string'] ]); $parent = $data['parent']; } - // TODO - Run import - // TODO - Validate again before - // TODO - Check permissions before (create for main item, create for children, create for related items [image, attachments]) + $entity = $this->imports->runImport($import, $parent); + if ($entity) { + $this->logActivity(ActivityType::IMPORT_RUN, $import); + return redirect($entity->getUrl()); + } // TODO - Redirect to result // TODO - Or redirect back with errors + return 'failed'; } /** diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index b94563545..d169d4845 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -2,9 +2,11 @@ namespace BookStack\Exports; +use BookStack\Entities\Models\Entity; use BookStack\Entities\Queries\EntityQueries; use BookStack\Exceptions\FileUploadException; use BookStack\Exceptions\ZipExportException; +use BookStack\Exceptions\ZipImportException; use BookStack\Exceptions\ZipValidationException; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; @@ -95,9 +97,9 @@ class ImportRepo } /** - * @throws ZipValidationException + * @throws ZipValidationException|ZipImportException */ - public function runImport(Import $import, ?string $parent = null) + public function runImport(Import $import, ?string $parent = null): ?Entity { $parentModel = null; if ($import->type === 'page' || $import->type === 'chapter') { diff --git a/app/Exports/ZipExports/ZipImportReferences.php b/app/Exports/ZipExports/ZipImportReferences.php index 8062886e5..3bce16bbb 100644 --- a/app/Exports/ZipExports/ZipImportReferences.php +++ b/app/Exports/ZipExports/ZipImportReferences.php @@ -110,7 +110,7 @@ class ZipImportReferences { foreach ($this->books as $book) { $exportBook = $this->zipExportBookMap[$book->id]; - $content = $exportBook->description_html || ''; + $content = $exportBook->description_html ?? ''; $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); $this->baseRepo->update($book, [ @@ -120,7 +120,7 @@ class ZipImportReferences foreach ($this->chapters as $chapter) { $exportChapter = $this->zipExportChapterMap[$chapter->id]; - $content = $exportChapter->description_html || ''; + $content = $exportChapter->description_html ?? ''; $parsed = $this->parser->parseReferences($content, $this->handleReference(...)); $this->baseRepo->update($chapter, [ diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 345c22be1..9f19f03e2 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -12,17 +12,22 @@ use BookStack\Entities\Repos\PageRepo; use BookStack\Exceptions\ZipExportException; use BookStack\Exceptions\ZipImportException; use BookStack\Exports\Import; +use BookStack\Exports\ZipExports\Models\ZipExportAttachment; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; +use BookStack\Exports\ZipExports\Models\ZipExportImage; use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Exports\ZipExports\Models\ZipExportTag; +use BookStack\Uploads\Attachment; +use BookStack\Uploads\AttachmentService; use BookStack\Uploads\FileStorage; +use BookStack\Uploads\Image; use BookStack\Uploads\ImageService; use Illuminate\Http\UploadedFile; class ZipImportRunner { - protected array $tempFilesToCleanup = []; // TODO + protected array $tempFilesToCleanup = []; public function __construct( protected FileStorage $storage, @@ -30,14 +35,19 @@ class ZipImportRunner protected ChapterRepo $chapterRepo, protected BookRepo $bookRepo, protected ImageService $imageService, + protected AttachmentService $attachmentService, protected ZipImportReferences $references, ) { } /** + * Run the import. + * Performs re-validation on zip, validation on parent provided, and permissions for importing + * the planned content, before running the import process. + * Returns the top-level entity item which was imported. * @throws ZipImportException */ - public function run(Import $import, ?Entity $parent = null): void + public function run(Import $import, ?Entity $parent = null): ?Entity { $zipPath = $this->getZipPath($import); $reader = new ZipExportReader($zipPath); @@ -63,8 +73,16 @@ class ZipImportRunner } $this->ensurePermissionsPermitImport($exportModel); + $entity = null; + + if ($exportModel instanceof ZipExportBook) { + $entity = $this->importBook($exportModel, $reader); + } else if ($exportModel instanceof ZipExportChapter) { + $entity = $this->importChapter($exportModel, $parent, $reader); + } else if ($exportModel instanceof ZipExportPage) { + $entity = $this->importPage($exportModel, $parent, $reader); + } - // TODO - Run import // TODO - In transaction? // TODO - Revert uploaded files if goes wrong // TODO - Attachments @@ -72,6 +90,23 @@ class ZipImportRunner // (Both listed/stored in references) $this->references->replaceReferences(); + + $reader->close(); + $this->cleanup(); + + dd('stop'); + + // TODO - Delete import/zip after import? + // Do this in parent repo? + + return $entity; + } + + protected function cleanup() + { + foreach ($this->tempFilesToCleanup as $file) { + unlink($file); + } } protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader): Book @@ -83,17 +118,26 @@ class ZipImportRunner 'tags' => $this->exportTagsToInputArray($exportBook->tags ?? []), ]); - // TODO - Parse/format description_html references - if ($book->cover) { $this->references->addImage($book->cover, null); } - // TODO - Pages - foreach ($exportBook->chapters as $exportChapter) { - $this->importChapter($exportChapter, $book, $reader); + $children = [ + ...$exportBook->chapters, + ...$exportBook->pages, + ]; + + usort($children, function (ZipExportPage|ZipExportChapter $a, ZipExportPage|ZipExportChapter $b) { + return ($a->priority ?? 0) - ($b->priority ?? 0); + }); + + foreach ($children as $child) { + if ($child instanceof ZipExportChapter) { + $this->importChapter($child, $book, $reader); + } else if ($child instanceof ZipExportPage) { + $this->importPage($child, $book, $reader); + } } - // TODO - Sort chapters/pages by order $this->references->addBook($book, $exportBook); @@ -108,17 +152,14 @@ class ZipImportRunner 'tags' => $this->exportTagsToInputArray($exportChapter->tags ?? []), ], $parent); - // TODO - Parse/format description_html references - $exportPages = $exportChapter->pages; usort($exportPages, function (ZipExportPage $a, ZipExportPage $b) { return ($a->priority ?? 0) - ($b->priority ?? 0); }); foreach ($exportPages as $exportPage) { - // + $this->importPage($exportPage, $chapter, $reader); } - // TODO - Pages $this->references->addChapter($chapter, $exportChapter); @@ -129,11 +170,13 @@ class ZipImportRunner { $page = $this->pageRepo->getNewDraftPage($parent); - // TODO - Import attachments - // TODO - Add attachment references - // TODO - Import images - // TODO - Add image references - // TODO - Parse/format HTML + foreach ($exportPage->attachments as $exportAttachment) { + $this->importAttachment($exportAttachment, $page, $reader); + } + + foreach ($exportPage->images as $exportImage) { + $this->importImage($exportImage, $page, $reader); + } $this->pageRepo->publishDraft($page, [ 'name' => $exportPage->name, @@ -147,6 +190,40 @@ class ZipImportRunner return $page; } + protected function importAttachment(ZipExportAttachment $exportAttachment, Page $page, ZipExportReader $reader): Attachment + { + if ($exportAttachment->file) { + $file = $this->zipFileToUploadedFile($exportAttachment->file, $reader); + $attachment = $this->attachmentService->saveNewUpload($file, $page->id); + $attachment->name = $exportAttachment->name; + $attachment->save(); + } else { + $attachment = $this->attachmentService->saveNewFromLink( + $exportAttachment->name, + $exportAttachment->link ?? '', + $page->id, + ); + } + + $this->references->addAttachment($attachment, $exportAttachment->id); + + return $attachment; + } + + protected function importImage(ZipExportImage $exportImage, Page $page, ZipExportReader $reader): Image + { + $file = $this->zipFileToUploadedFile($exportImage->file, $reader); + $image = $this->imageService->saveNewFromUpload( + $file, + $exportImage->type, + $page->id, + ); + + $this->references->addImage($image, $exportImage->id); + + return $image; + } + protected function exportTagsToInputArray(array $exportTags): array { $tags = []; @@ -235,7 +312,7 @@ class ZipImportRunner } if (count($attachments) > 0) { - if (userCan('attachment-create-all')) { + if (!userCan('attachment-create-all')) { $errors[] = 'You are lacking the required permissions to create attachments.'; } } @@ -257,6 +334,8 @@ class ZipImportRunner stream_copy_to_stream($stream, $tempFile); fclose($tempFile); + $this->tempFilesToCleanup[] = $tempFilePath; + return $tempFilePath; } } From b7476a9e7fc27c27342a0a155ab256a93f19981e Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Thu, 14 Nov 2024 15:59:15 +0000 Subject: [PATCH 29/41] ZIP Import: Finished base import process & error handling Added file creation reverting and DB rollback on error. Added error display on failed import. Extracted likely shown import form/error text to translation files. --- app/Exports/Controllers/ImportController.php | 25 +++---- app/Exports/ImportRepo.php | 26 ++++++- .../ZipExports/ZipImportReferences.php | 17 +++++ app/Exports/ZipExports/ZipImportRunner.php | 71 +++++++++++-------- app/Uploads/AttachmentService.php | 2 +- app/Uploads/ImageService.php | 12 +++- lang/en/entities.php | 3 + lang/en/errors.php | 6 ++ resources/views/exports/import-show.blade.php | 49 ++++++++----- .../views/exports/parts/import.blade.php | 4 +- 10 files changed, 146 insertions(+), 69 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index 4d2c83090..d8dceed2f 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -4,7 +4,7 @@ declare(strict_types=1); namespace BookStack\Exports\Controllers; -use BookStack\Activity\ActivityType; +use BookStack\Exceptions\ZipImportException; use BookStack\Exceptions\ZipValidationException; use BookStack\Exports\ImportRepo; use BookStack\Http\Controller; @@ -48,12 +48,9 @@ class ImportController extends Controller try { $import = $this->imports->storeFromUpload($file); } catch (ZipValidationException $exception) { - session()->flash('validation_errors', $exception->errors); - return redirect('/import'); + return redirect('/import')->with('validation_errors', $exception->errors); } - $this->logActivity(ActivityType::IMPORT_CREATE, $import); - return redirect($import->getUrl()); } @@ -80,20 +77,20 @@ class ImportController extends Controller $parent = null; if ($import->type === 'page' || $import->type === 'chapter') { + session()->setPreviousUrl($import->getUrl()); $data = $this->validate($request, [ - 'parent' => ['required', 'string'] + 'parent' => ['required', 'string'], ]); $parent = $data['parent']; } - $entity = $this->imports->runImport($import, $parent); - if ($entity) { - $this->logActivity(ActivityType::IMPORT_RUN, $import); - return redirect($entity->getUrl()); + try { + $entity = $this->imports->runImport($import, $parent); + } catch (ZipImportException $exception) { + return redirect($import->getUrl())->with('import_errors', $exception->errors); } - // TODO - Redirect to result - // TODO - Or redirect back with errors - return 'failed'; + + return redirect($entity->getUrl()); } /** @@ -104,8 +101,6 @@ class ImportController extends Controller $import = $this->imports->findVisible($id); $this->imports->deleteImport($import); - $this->logActivity(ActivityType::IMPORT_DELETE, $import); - return redirect('/import'); } } diff --git a/app/Exports/ImportRepo.php b/app/Exports/ImportRepo.php index d169d4845..f72386c47 100644 --- a/app/Exports/ImportRepo.php +++ b/app/Exports/ImportRepo.php @@ -2,6 +2,7 @@ namespace BookStack\Exports; +use BookStack\Activity\ActivityType; use BookStack\Entities\Models\Entity; use BookStack\Entities\Queries\EntityQueries; use BookStack\Exceptions\FileUploadException; @@ -14,8 +15,10 @@ use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Exports\ZipExports\ZipExportReader; use BookStack\Exports\ZipExports\ZipExportValidator; use BookStack\Exports\ZipExports\ZipImportRunner; +use BookStack\Facades\Activity; use BookStack\Uploads\FileStorage; use Illuminate\Database\Eloquent\Collection; +use Illuminate\Support\Facades\DB; use Symfony\Component\HttpFoundation\File\UploadedFile; class ImportRepo @@ -93,25 +96,42 @@ class ImportRepo $import->path = $path; $import->save(); + Activity::add(ActivityType::IMPORT_CREATE, $import); + return $import; } /** - * @throws ZipValidationException|ZipImportException + * @throws ZipImportException */ - public function runImport(Import $import, ?string $parent = null): ?Entity + public function runImport(Import $import, ?string $parent = null): Entity { $parentModel = null; if ($import->type === 'page' || $import->type === 'chapter') { $parentModel = $parent ? $this->entityQueries->findVisibleByStringIdentifier($parent) : null; } - return $this->importer->run($import, $parentModel); + DB::beginTransaction(); + try { + $model = $this->importer->run($import, $parentModel); + } catch (ZipImportException $e) { + DB::rollBack(); + $this->importer->revertStoredFiles(); + throw $e; + } + + DB::commit(); + $this->deleteImport($import); + Activity::add(ActivityType::IMPORT_RUN, $import); + + return $model; } public function deleteImport(Import $import): void { $this->storage->delete($import->path); $import->delete(); + + Activity::add(ActivityType::IMPORT_DELETE, $import); } } diff --git a/app/Exports/ZipExports/ZipImportReferences.php b/app/Exports/ZipExports/ZipImportReferences.php index 3bce16bbb..b23d5e72b 100644 --- a/app/Exports/ZipExports/ZipImportReferences.php +++ b/app/Exports/ZipExports/ZipImportReferences.php @@ -139,4 +139,21 @@ class ZipImportReferences ]); } } + + + /** + * @return Image[] + */ + public function images(): array + { + return $this->images; + } + + /** + * @return Attachment[] + */ + public function attachments(): array + { + return $this->attachments; + } } diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 9f19f03e2..c5b9da319 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -47,14 +47,17 @@ class ZipImportRunner * Returns the top-level entity item which was imported. * @throws ZipImportException */ - public function run(Import $import, ?Entity $parent = null): ?Entity + public function run(Import $import, ?Entity $parent = null): Entity { $zipPath = $this->getZipPath($import); $reader = new ZipExportReader($zipPath); $errors = (new ZipExportValidator($reader))->validate(); if ($errors) { - throw new ZipImportException(["ZIP failed to validate"]); + throw new ZipImportException([ + trans('errors.import_validation_failed'), + ...$errors, + ]); } try { @@ -65,15 +68,14 @@ class ZipImportRunner // Validate parent type if ($exportModel instanceof ZipExportBook && ($parent !== null)) { - throw new ZipImportException(["Must not have a parent set for a Book import"]); - } else if ($exportModel instanceof ZipExportChapter && (!$parent instanceof Book)) { - throw new ZipImportException(["Parent book required for chapter import"]); + throw new ZipImportException(["Must not have a parent set for a Book import."]); + } else if ($exportModel instanceof ZipExportChapter && !($parent instanceof Book)) { + throw new ZipImportException(["Parent book required for chapter import."]); } else if ($exportModel instanceof ZipExportPage && !($parent instanceof Book || $parent instanceof Chapter)) { - throw new ZipImportException(["Parent book or chapter required for page import"]); + throw new ZipImportException(["Parent book or chapter required for page import."]); } - $this->ensurePermissionsPermitImport($exportModel); - $entity = null; + $this->ensurePermissionsPermitImport($exportModel, $parent); if ($exportModel instanceof ZipExportBook) { $entity = $this->importBook($exportModel, $reader); @@ -81,32 +83,46 @@ class ZipImportRunner $entity = $this->importChapter($exportModel, $parent, $reader); } else if ($exportModel instanceof ZipExportPage) { $entity = $this->importPage($exportModel, $parent, $reader); + } else { + throw new ZipImportException(['No importable data found in import data.']); } - // TODO - In transaction? - // TODO - Revert uploaded files if goes wrong - // TODO - Attachments - // TODO - Images - // (Both listed/stored in references) - $this->references->replaceReferences(); $reader->close(); $this->cleanup(); - dd('stop'); - - // TODO - Delete import/zip after import? - // Do this in parent repo? - return $entity; } - protected function cleanup() + /** + * Revert any files which have been stored during this import process. + * Considers files only, and avoids the database under the + * assumption that the database may already have been + * reverted as part of a transaction rollback. + */ + public function revertStoredFiles(): void + { + foreach ($this->references->images() as $image) { + $this->imageService->destroyFileAtPath($image->type, $image->path); + } + + foreach ($this->references->attachments() as $attachment) { + if (!$attachment->external) { + $this->attachmentService->deleteFileInStorage($attachment); + } + } + + $this->cleanup(); + } + + protected function cleanup(): void { foreach ($this->tempFilesToCleanup as $file) { unlink($file); } + + $this->tempFilesToCleanup = []; } protected function importBook(ZipExportBook $exportBook, ZipExportReader $reader): Book @@ -256,9 +272,6 @@ class ZipImportRunner { $errors = []; - // TODO - Extract messages to language files - // TODO - Ensure these are shown to users on failure - $chapters = []; $pages = []; $images = []; @@ -266,7 +279,7 @@ class ZipImportRunner if ($exportModel instanceof ZipExportBook) { if (!userCan('book-create-all')) { - $errors[] = 'You are lacking the required permission to create books.'; + $errors[] = trans('errors.import_perms_books'); } array_push($pages, ...$exportModel->pages); array_push($chapters, ...$exportModel->chapters); @@ -283,7 +296,7 @@ class ZipImportRunner if (count($chapters) > 0) { $permission = 'chapter-create' . ($parent ? '' : '-all'); if (!userCan($permission, $parent)) { - $errors[] = 'You are lacking the required permission to create chapters.'; + $errors[] = trans('errors.import_perms_chapters'); } } @@ -295,25 +308,25 @@ class ZipImportRunner if (count($pages) > 0) { if ($parent) { if (!userCan('page-create', $parent)) { - $errors[] = 'You are lacking the required permission to create pages.'; + $errors[] = trans('errors.import_perms_pages'); } } else { $hasPermission = userCan('page-create-all') || userCan('page-create-own'); if (!$hasPermission) { - $errors[] = 'You are lacking the required permission to create pages.'; + $errors[] = trans('errors.import_perms_pages'); } } } if (count($images) > 0) { if (!userCan('image-create-all')) { - $errors[] = 'You are lacking the required permissions to create images.'; + $errors[] = trans('errors.import_perms_images'); } } if (count($attachments) > 0) { if (!userCan('attachment-create-all')) { - $errors[] = 'You are lacking the required permissions to create attachments.'; + $errors[] = trans('errors.import_perms_attachments'); } } diff --git a/app/Uploads/AttachmentService.php b/app/Uploads/AttachmentService.php index fa53c4ae4..033f23341 100644 --- a/app/Uploads/AttachmentService.php +++ b/app/Uploads/AttachmentService.php @@ -151,7 +151,7 @@ class AttachmentService * Delete a file from the filesystem it sits on. * Cleans any empty leftover folders. */ - protected function deleteFileInStorage(Attachment $attachment): void + public function deleteFileInStorage(Attachment $attachment): void { $this->storage->delete($attachment->path); } diff --git a/app/Uploads/ImageService.php b/app/Uploads/ImageService.php index e501cc7b1..5c455cf86 100644 --- a/app/Uploads/ImageService.php +++ b/app/Uploads/ImageService.php @@ -153,11 +153,19 @@ class ImageService */ public function destroy(Image $image): void { - $disk = $this->storage->getDisk($image->type); - $disk->destroyAllMatchingNameFromPath($image->path); + $this->destroyFileAtPath($image->type, $image->path); $image->delete(); } + /** + * Destroy the underlying image file at the given path. + */ + public function destroyFileAtPath(string $type, string $path): void + { + $disk = $this->storage->getDisk($type); + $disk->destroyAllMatchingNameFromPath($path); + } + /** * Delete gallery and drawings that are not within HTML content of pages or page revisions. * Checks based off of only the image name. diff --git a/lang/en/entities.php b/lang/en/entities.php index ae1c1e8d4..26a563a7e 100644 --- a/lang/en/entities.php +++ b/lang/en/entities.php @@ -52,6 +52,7 @@ return [ 'import_pending_none' => 'No imports have been started.', 'import_continue' => 'Continue Import', 'import_continue_desc' => 'Review the content due to be imported from the uploaded ZIP file. When ready, run the import to add its contents to this system. The uploaded ZIP import file will be automatically removed on successful import.', + 'import_details' => 'Import Details', 'import_run' => 'Run Import', 'import_size' => ':size Import ZIP Size', 'import_uploaded_at' => 'Uploaded :relativeTime', @@ -60,6 +61,8 @@ return [ 'import_location_desc' => 'Select a target location for your imported content. You\'ll need the relevant permissions to create within the location you choose.', 'import_delete_confirm' => 'Are you sure you want to delete this import?', 'import_delete_desc' => 'This will delete the uploaded import ZIP file, and cannot be undone.', + 'import_errors' => 'Import Errors', + 'import_errors_desc' => 'The follow errors occurred during the import attempt:', // Permissions and restrictions 'permissions' => 'Permissions', diff --git a/lang/en/errors.php b/lang/en/errors.php index 3f2f30331..ced80a32c 100644 --- a/lang/en/errors.php +++ b/lang/en/errors.php @@ -109,6 +109,12 @@ return [ 'import_zip_cant_read' => 'Could not read ZIP file.', 'import_zip_cant_decode_data' => 'Could not find and decode ZIP data.json content.', 'import_zip_no_data' => 'ZIP file data has no expected book, chapter or page content.', + 'import_validation_failed' => 'Import ZIP failed to validate with errors:', + 'import_perms_books' => 'You are lacking the required permissions to create books.', + 'import_perms_chapters' => 'You are lacking the required permissions to create chapters.', + 'import_perms_pages' => 'You are lacking the required permissions to create pages.', + 'import_perms_images' => 'You are lacking the required permissions to create images.', + 'import_perms_attachments' => 'You are lacking the required permission to create attachments.', // API errors 'api_no_authorization_found' => 'No authorization token found on the request', diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php index 40867377f..e4f199aa2 100644 --- a/resources/views/exports/import-show.blade.php +++ b/resources/views/exports/import-show.blade.php @@ -7,8 +7,19 @@

    {{ trans('entities.import_continue') }}

    {{ trans('entities.import_continue_desc') }}

    + @if(session()->has('import_errors')) +
    + +

    {{ trans('entities.import_errors_desc') }}

    + @foreach(session()->get('import_errors') ?? [] as $error) +

    {{ $error }}

    + @endforeach +
    +
    + @endif +
    - +
    @include('exports.parts.import-item', ['type' => $import->type, 'model' => $data]) @@ -34,32 +45,36 @@ @if($import->type === 'page' || $import->type === 'chapter')
    -

    {{ trans('entities.import_location_desc') }}

    +

    {{ trans('entities.import_location_desc') }}

    + @if($errors->has('parent')) +
    + @include('form.errors', ['name' => 'parent']) +
    + @endif @include('entities.selector', [ 'name' => 'parent', 'entityTypes' => $import->type === 'page' ? 'chapter,book' : 'book', 'entityPermission' => "{$import->type}-create", 'selectorSize' => 'compact small', ]) - @include('form.errors', ['name' => 'parent']) @endif - -
    - {{ trans('common.cancel') }} -
    - -
    -

    {{ trans('entities.import_delete_confirm') }}

    -

    {{ trans('entities.import_delete_desc') }}

    - +
    + {{ trans('common.cancel') }} +
    + +
    +

    {{ trans('entities.import_delete_confirm') }}

    +

    {{ trans('entities.import_delete_desc') }}

    + +
    +
    - -
    +
    diff --git a/resources/views/exports/parts/import.blade.php b/resources/views/exports/parts/import.blade.php index fd53095a4..2f7659c46 100644 --- a/resources/views/exports/parts/import.blade.php +++ b/resources/views/exports/parts/import.blade.php @@ -4,7 +4,7 @@ class="text-{{ $import->type }}">@icon($import->type) {{ $import->name }}
    -
    {{ $import->getSizeString() }}
    -
    @icon('time'){{ $import->created_at->diffForHumans() }}
    +
    {{ $import->getSizeString() }}
    +
    @icon('time'){{ $import->created_at->diffForHumans() }}
    \ No newline at end of file From 7681e32dca6cb7d06c2d196bf46239a41a86852c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 16 Nov 2024 13:57:41 +0000 Subject: [PATCH 30/41] ZIP Imports: Added high level import run tests --- app/Exports/Controllers/ImportController.php | 4 +- database/factories/Exports/ImportFactory.php | 2 +- tests/Exports/ZipImportRunnerTest.php | 21 +++ tests/Exports/ZipImportTest.php | 139 ++++++++++++++++--- tests/Exports/ZipTestHelper.php | 47 +++++++ 5 files changed, 195 insertions(+), 18 deletions(-) create mode 100644 tests/Exports/ZipImportRunnerTest.php create mode 100644 tests/Exports/ZipTestHelper.php diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index d8dceed2f..a20c341fb 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -70,9 +70,11 @@ class ImportController extends Controller ]); } + /** + * Run the import process against an uploaded import ZIP. + */ public function run(int $id, Request $request) { - // TODO - Test access/visibility $import = $this->imports->findVisible($id); $parent = null; diff --git a/database/factories/Exports/ImportFactory.php b/database/factories/Exports/ImportFactory.php index 74a2bcd65..5d0b4f892 100644 --- a/database/factories/Exports/ImportFactory.php +++ b/database/factories/Exports/ImportFactory.php @@ -21,7 +21,7 @@ class ImportFactory extends Factory public function definition(): array { return [ - 'path' => 'uploads/imports/' . Str::random(10) . '.zip', + 'path' => 'uploads/files/imports/' . Str::random(10) . '.zip', 'name' => $this->faker->words(3, true), 'type' => 'book', 'metadata' => '{"name": "My book"}', diff --git a/tests/Exports/ZipImportRunnerTest.php b/tests/Exports/ZipImportRunnerTest.php new file mode 100644 index 000000000..7bdd8ecbb --- /dev/null +++ b/tests/Exports/ZipImportRunnerTest.php @@ -0,0 +1,21 @@ +runner = app()->make(ZipImportRunner::class); + } + + // TODO - Test full book import + // TODO - Test full chapter import + // TODO - Test full page import +} diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php index 2b40100aa..3644e9bdc 100644 --- a/tests/Exports/ZipImportTest.php +++ b/tests/Exports/ZipImportTest.php @@ -3,6 +3,7 @@ namespace Tests\Exports; use BookStack\Activity\ActivityType; +use BookStack\Entities\Models\Book; use BookStack\Exports\Import; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; @@ -91,7 +92,7 @@ class ZipImportTest extends TestCase public function test_error_shown_if_no_importable_key() { $this->asAdmin(); - $resp = $this->runImportFromFile($this->zipUploadFromData([ + $resp = $this->runImportFromFile(ZipTestHelper::zipUploadFromData([ 'instance' => [] ])); @@ -103,7 +104,7 @@ class ZipImportTest extends TestCase public function test_zip_data_validation_messages_shown() { $this->asAdmin(); - $resp = $this->runImportFromFile($this->zipUploadFromData([ + $resp = $this->runImportFromFile(ZipTestHelper::zipUploadFromData([ 'book' => [ 'id' => 4, 'pages' => [ @@ -154,7 +155,7 @@ class ZipImportTest extends TestCase ], ]; - $resp = $this->runImportFromFile($this->zipUploadFromData($data)); + $resp = $this->runImportFromFile(ZipTestHelper::zipUploadFromData($data)); $this->assertDatabaseHas('imports', [ 'name' => 'My great book name', @@ -217,7 +218,7 @@ class ZipImportTest extends TestCase public function test_import_delete() { $this->asAdmin(); - $this->runImportFromFile($this->zipUploadFromData([ + $this->runImportFromFile(ZipTestHelper::zipUploadFromData([ 'book' => [ 'name' => 'My great book name' ], @@ -262,20 +263,126 @@ class ZipImportTest extends TestCase $this->delete("/import/{$adminImport->id}")->assertRedirect('/import'); } + public function test_run_simple_success_scenario() + { + $import = ZipTestHelper::importFromData([], [ + 'book' => [ + 'name' => 'My imported book', + 'pages' => [ + [ + 'name' => 'My imported book page', + 'html' => '

    Hello there from child page!

    ' + ] + ], + ] + ]); + + $resp = $this->asAdmin()->post("/import/{$import->id}"); + $book = Book::query()->where('name', '=', 'My imported book')->latest()->first(); + $resp->assertRedirect($book->getUrl()); + + $resp = $this->followRedirects($resp); + $resp->assertSee('My imported book page'); + $resp->assertSee('Hello there from child page!'); + + $this->assertDatabaseMissing('imports', ['id' => $import->id]); + $this->assertFileDoesNotExist(storage_path($import->path)); + $this->assertActivityExists(ActivityType::IMPORT_RUN, null, $import->logDescriptor()); + } + + public function test_import_run_access_limited() + { + $user = $this->users->editor(); + $admin = $this->users->admin(); + $userImport = Import::factory()->create(['name' => 'MySuperUserImport', 'created_by' => $user->id]); + $adminImport = Import::factory()->create(['name' => 'MySuperAdminImport', 'created_by' => $admin->id]); + $this->actingAs($user); + + $this->post("/import/{$userImport->id}")->assertRedirect('/'); + $this->post("/import/{$adminImport->id}")->assertRedirect('/'); + + $this->permissions->grantUserRolePermissions($user, ['content-import']); + + $this->post("/import/{$userImport->id}")->assertRedirect($userImport->getUrl()); // Getting validation response instead of access issue response + $this->post("/import/{$adminImport->id}")->assertStatus(404); + + $this->permissions->grantUserRolePermissions($user, ['settings-manage']); + + $this->post("/import/{$adminImport->id}")->assertRedirect($adminImport->getUrl()); // Getting validation response instead of access issue response + } + + public function test_run_revalidates_content() + { + $import = ZipTestHelper::importFromData([], [ + 'book' => [ + 'id' => 'abc', + ] + ]); + + $resp = $this->asAdmin()->post("/import/{$import->id}"); + $resp->assertRedirect($import->getUrl()); + + $resp = $this->followRedirects($resp); + $resp->assertSeeText('The name field is required.'); + $resp->assertSeeText('The id must be an integer.'); + } + + public function test_run_checks_permissions_on_import() + { + $viewer = $this->users->viewer(); + $this->permissions->grantUserRolePermissions($viewer, ['content-import']); + $import = ZipTestHelper::importFromData(['created_by' => $viewer->id], [ + 'book' => ['name' => 'My import book'], + ]); + + $resp = $this->asViewer()->post("/import/{$import->id}"); + $resp->assertRedirect($import->getUrl()); + + $resp = $this->followRedirects($resp); + $resp->assertSeeText('You are lacking the required permissions to create books.'); + } + + public function test_run_requires_parent_for_chapter_and_page_imports() + { + $book = $this->entities->book(); + $pageImport = ZipTestHelper::importFromData([], [ + 'page' => ['name' => 'My page', 'html' => '

    page test!

    '], + ]); + $chapterImport = ZipTestHelper::importFromData([], [ + 'chapter' => ['name' => 'My chapter'], + ]); + + $resp = $this->asAdmin()->post("/import/{$pageImport->id}"); + $resp->assertRedirect($pageImport->getUrl()); + $this->followRedirects($resp)->assertSee('The parent field is required.'); + + $resp = $this->asAdmin()->post("/import/{$pageImport->id}", ['parent' => "book:{$book->id}"]); + $resp->assertRedirectContains($book->getUrl()); + + $resp = $this->asAdmin()->post("/import/{$chapterImport->id}"); + $resp->assertRedirect($chapterImport->getUrl()); + $this->followRedirects($resp)->assertSee('The parent field is required.'); + + $resp = $this->asAdmin()->post("/import/{$chapterImport->id}", ['parent' => "book:{$book->id}"]); + $resp->assertRedirectContains($book->getUrl()); + } + + public function test_run_validates_correct_parent_type() + { + $chapter = $this->entities->chapter(); + $import = ZipTestHelper::importFromData([], [ + 'chapter' => ['name' => 'My chapter'], + ]); + + $resp = $this->asAdmin()->post("/import/{$import->id}", ['parent' => "chapter:{$chapter->id}"]); + $resp->assertRedirect($import->getUrl()); + + $resp = $this->followRedirects($resp); + $resp->assertSee('Parent book required for chapter import.'); + } + protected function runImportFromFile(UploadedFile $file): TestResponse { return $this->call('POST', '/import', [], [], ['file' => $file]); } - - protected function zipUploadFromData(array $data): UploadedFile - { - $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); - - $zip = new ZipArchive(); - $zip->open($zipFile, ZipArchive::CREATE); - $zip->addFromString('data.json', json_encode($data)); - $zip->close(); - - return new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); - } } diff --git a/tests/Exports/ZipTestHelper.php b/tests/Exports/ZipTestHelper.php new file mode 100644 index 000000000..3a9b34354 --- /dev/null +++ b/tests/Exports/ZipTestHelper.php @@ -0,0 +1,47 @@ +create($importData); + $zip = static::zipUploadFromData($zipData); + rename($zip->getRealPath(), storage_path($import->path)); + + return $import; + } + + public static function deleteZipForImport(Import $import): void + { + $path = storage_path($import->path); + if (file_exists($path)) { + unlink($path); + } + } + + public static function zipUploadFromData(array $data): UploadedFile + { + $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); + + $zip = new ZipArchive(); + $zip->open($zipFile, ZipArchive::CREATE); + $zip->addFromString('data.json', json_encode($data)); + $zip->close(); + + return new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); + } +} From 8645aeaa4a914c5ee7e0d07a9202b8812aefcafe Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 16 Nov 2024 16:12:45 +0000 Subject: [PATCH 31/41] ZIP Imports: Started testing core import logic Fixed image size handling, and lack of attachment reference replacements during testing. --- .../ZipExports/ZipImportReferences.php | 4 +- app/Exports/ZipExports/ZipImportRunner.php | 4 + app/Uploads/ImageService.php | 5 +- tests/Exports/ZipImportRunnerTest.php | 152 ++++++++++++++++++ tests/Exports/ZipTestHelper.php | 11 +- 5 files changed, 170 insertions(+), 6 deletions(-) diff --git a/app/Exports/ZipExports/ZipImportReferences.php b/app/Exports/ZipExports/ZipImportReferences.php index b23d5e72b..da0581df6 100644 --- a/app/Exports/ZipExports/ZipImportReferences.php +++ b/app/Exports/ZipExports/ZipImportReferences.php @@ -97,10 +97,12 @@ class ZipImportReferences } else if ($model instanceof Image) { if ($model->type === 'gallery') { $this->imageResizer->loadGalleryThumbnailsForImage($model, false); - return $model->thumbs['gallery'] ?? $model->url; + return $model->thumbs['display'] ?? $model->url; } return $model->url; + } else if ($model instanceof Attachment) { + return $model->getUrl(false); } return null; diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index c5b9da319..27d859e59 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -233,6 +233,10 @@ class ZipImportRunner $file, $exportImage->type, $page->id, + null, + null, + true, + $exportImage->name, ); $this->references->addImage($image, $exportImage->id); diff --git a/app/Uploads/ImageService.php b/app/Uploads/ImageService.php index 5c455cf86..038e6aa41 100644 --- a/app/Uploads/ImageService.php +++ b/app/Uploads/ImageService.php @@ -33,9 +33,10 @@ class ImageService int $uploadedTo = 0, int $resizeWidth = null, int $resizeHeight = null, - bool $keepRatio = true + bool $keepRatio = true, + string $imageName = '', ): Image { - $imageName = $uploadedFile->getClientOriginalName(); + $imageName = $imageName ?: $uploadedFile->getClientOriginalName(); $imageData = file_get_contents($uploadedFile->getRealPath()); if ($resizeWidth !== null || $resizeHeight !== null) { diff --git a/tests/Exports/ZipImportRunnerTest.php b/tests/Exports/ZipImportRunnerTest.php index 7bdd8ecbb..f07b3f41b 100644 --- a/tests/Exports/ZipImportRunnerTest.php +++ b/tests/Exports/ZipImportRunnerTest.php @@ -2,7 +2,10 @@ namespace Tests\Exports; +use BookStack\Entities\Models\Book; +use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\ZipImportRunner; +use BookStack\Uploads\Image; use Tests\TestCase; class ZipImportRunnerTest extends TestCase @@ -15,6 +18,155 @@ class ZipImportRunnerTest extends TestCase $this->runner = app()->make(ZipImportRunner::class); } + public function test_book_import() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $testFilePath = $this->files->testFilePath('test-file.txt'); + $import = ZipTestHelper::importFromData([], [ + 'book' => [ + 'id' => 5, + 'name' => 'Import test', + 'cover' => 'book_cover_image', + 'description_html' => '

    Link to chapter page

    ', + 'tags' => [ + ['name' => 'Animal', 'value' => 'Cat'], + ['name' => 'Category', 'value' => 'Test'], + ], + 'chapters' => [ + [ + 'id' => 6, + 'name' => 'Chapter A', + 'description_html' => '

    Link to book

    ', + 'priority' => 1, + 'tags' => [ + ['name' => 'Reviewed'], + ['name' => 'Category', 'value' => 'Test Chapter'], + ], + 'pages' => [ + [ + 'id' => 3, + 'name' => 'Page A', + 'priority' => 6, + 'html' => ' +

    Link to self

    +

    Link to cat image

    +

    Link to text attachment

    ', + 'tags' => [ + ['name' => 'Unreviewed'], + ], + 'attachments' => [ + [ + 'id' => 4, + 'name' => 'Text attachment', + 'file' => 'file_attachment' + ], + [ + 'name' => 'Cats', + 'link' => 'https://example.com/cats', + ] + ], + 'images' => [ + [ + 'id' => 1, + 'name' => 'Cat', + 'type' => 'gallery', + 'file' => 'cat_image' + ], + [ + 'id' => 2, + 'name' => 'Dog Drawing', + 'type' => 'drawio', + 'file' => 'dog_image' + ] + ], + ], + ], + ], + [ + 'name' => 'Chapter child B', + 'priority' => 5, + ] + ], + 'pages' => [ + [ + 'name' => 'Page C', + 'markdown' => '[Link to text]([[bsexport:attachment:4]]?scale=big)', + 'priority' => 3, + ] + ], + ], + ], [ + 'book_cover_image' => $testImagePath, + 'file_attachment' => $testFilePath, + 'cat_image' => $testImagePath, + 'dog_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Book $book */ + $book = $this->runner->run($import); + + // Book checks + $this->assertEquals('Import test', $book->name); + $this->assertFileExists(public_path($book->cover->path)); + $this->assertCount(2, $book->tags); + $this->assertEquals('Cat', $book->tags()->first()->value); + $this->assertCount(2, $book->chapters); + $this->assertEquals(1, $book->directPages()->count()); + + // Chapter checks + $chapterA = $book->chapters()->where('name', 'Chapter A')->first(); + $this->assertCount(2, $chapterA->tags); + $firstChapterTag = $chapterA->tags()->first(); + $this->assertEquals('Reviewed', $firstChapterTag->name); + $this->assertEquals('', $firstChapterTag->value); + $this->assertCount(1, $chapterA->pages); + + // Page checks + /** @var Page $pageA */ + $pageA = $chapterA->pages->first(); + $this->assertEquals('Page A', $pageA->name); + $this->assertCount(1, $pageA->tags); + $firstPageTag = $pageA->tags()->first(); + $this->assertEquals('Unreviewed', $firstPageTag->name); + $this->assertCount(2, $pageA->attachments); + $firstAttachment = $pageA->attachments->first(); + $this->assertEquals('Text attachment', $firstAttachment->name); + $this->assertFileEquals($testFilePath, storage_path($firstAttachment->path)); + $this->assertFalse($firstAttachment->external); + $secondAttachment = $pageA->attachments->last(); + $this->assertEquals('Cats', $secondAttachment->name); + $this->assertEquals('https://example.com/cats', $secondAttachment->path); + $this->assertTrue($secondAttachment->external); + $pageAImages = Image::where('uploaded_to', '=', $pageA->id)->whereIn('type', ['gallery', 'drawio'])->get(); + $this->assertCount(2, $pageAImages); + $this->assertEquals('Cat', $pageAImages[0]->name); + $this->assertEquals('gallery', $pageAImages[0]->type); + $this->assertFileEquals($testImagePath, public_path($pageAImages[0]->path)); + $this->assertEquals('Dog Drawing', $pageAImages[1]->name); + $this->assertEquals('drawio', $pageAImages[1]->type); + + // Book order check + $children = $book->getDirectVisibleChildren()->values()->all(); + $this->assertEquals($children[0]->name, 'Chapter A'); + $this->assertEquals($children[1]->name, 'Page C'); + $this->assertEquals($children[2]->name, 'Chapter child B'); + + // Reference checks + $textAttachmentUrl = $firstAttachment->getUrl(); + $this->assertStringContainsString($pageA->getUrl(), $book->description_html); + $this->assertStringContainsString($book->getUrl(), $chapterA->description_html); + $this->assertStringContainsString($pageA->getUrl(), $pageA->html); + $this->assertStringContainsString($pageAImages[0]->getThumb(1680, null, true), $pageA->html); + $this->assertStringContainsString($firstAttachment->getUrl(), $pageA->html); + + // Reference in converted markdown + $pageC = $children[1]; + $this->assertStringContainsString("href=\"{$textAttachmentUrl}?scale=big\"", $pageC->html); + + ZipTestHelper::deleteZipForImport($import); + } + // TODO - Test full book import // TODO - Test full chapter import // TODO - Test full page import diff --git a/tests/Exports/ZipTestHelper.php b/tests/Exports/ZipTestHelper.php index 3a9b34354..2196f361c 100644 --- a/tests/Exports/ZipTestHelper.php +++ b/tests/Exports/ZipTestHelper.php @@ -8,7 +8,7 @@ use ZipArchive; class ZipTestHelper { - public static function importFromData(array $importData, array $zipData): Import + public static function importFromData(array $importData, array $zipData, array $files = []): Import { if (isset($zipData['book'])) { $importData['type'] = 'book'; @@ -19,7 +19,7 @@ class ZipTestHelper } $import = Import::factory()->create($importData); - $zip = static::zipUploadFromData($zipData); + $zip = static::zipUploadFromData($zipData, $files); rename($zip->getRealPath(), storage_path($import->path)); return $import; @@ -33,13 +33,18 @@ class ZipTestHelper } } - public static function zipUploadFromData(array $data): UploadedFile + public static function zipUploadFromData(array $data, array $files = []): UploadedFile { $zipFile = tempnam(sys_get_temp_dir(), 'bstest-'); $zip = new ZipArchive(); $zip->open($zipFile, ZipArchive::CREATE); $zip->addFromString('data.json', json_encode($data)); + + foreach ($files as $name => $file) { + $zip->addFile($file, "files/$name"); + } + $zip->close(); return new UploadedFile($zipFile, 'upload.zip', 'application/zip', null, true); From c2c64e207f89567350eab4b40b725e8d042c9654 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Sat, 16 Nov 2024 19:52:20 +0000 Subject: [PATCH 32/41] ZIP Imports: Covered import runner with further testing --- tests/Exports/ZipImportRunnerTest.php | 194 +++++++++++++++++++++++++- 1 file changed, 191 insertions(+), 3 deletions(-) diff --git a/tests/Exports/ZipImportRunnerTest.php b/tests/Exports/ZipImportRunnerTest.php index f07b3f41b..c833fadda 100644 --- a/tests/Exports/ZipImportRunnerTest.php +++ b/tests/Exports/ZipImportRunnerTest.php @@ -3,6 +3,7 @@ namespace Tests\Exports; use BookStack\Entities\Models\Book; +use BookStack\Entities\Models\Chapter; use BookStack\Entities\Models\Page; use BookStack\Exports\ZipExports\ZipImportRunner; use BookStack\Uploads\Image; @@ -167,7 +168,194 @@ class ZipImportRunnerTest extends TestCase ZipTestHelper::deleteZipForImport($import); } - // TODO - Test full book import - // TODO - Test full chapter import - // TODO - Test full page import + public function test_chapter_import() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $testFilePath = $this->files->testFilePath('test-file.txt'); + $parent = $this->entities->book(); + + $import = ZipTestHelper::importFromData([], [ + 'chapter' => [ + 'id' => 6, + 'name' => 'Chapter A', + 'description_html' => '

    Link to page

    ', + 'priority' => 1, + 'tags' => [ + ['name' => 'Reviewed', 'value' => '2024'], + ], + 'pages' => [ + [ + 'id' => 3, + 'name' => 'Page A', + 'priority' => 6, + 'html' => '

    Link to chapter

    +

    Link to dog drawing

    +

    Link to text attachment

    ', + 'tags' => [ + ['name' => 'Unreviewed'], + ], + 'attachments' => [ + [ + 'id' => 4, + 'name' => 'Text attachment', + 'file' => 'file_attachment' + ] + ], + 'images' => [ + [ + 'id' => 2, + 'name' => 'Dog Drawing', + 'type' => 'drawio', + 'file' => 'dog_image' + ] + ], + ], + [ + 'name' => 'Page B', + 'markdown' => '[Link to page A]([[bsexport:page:3]])', + 'priority' => 9, + ], + ], + ], + ], [ + 'file_attachment' => $testFilePath, + 'dog_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Chapter $chapter */ + $chapter = $this->runner->run($import, $parent); + + // Chapter checks + $this->assertEquals('Chapter A', $chapter->name); + $this->assertEquals($parent->id, $chapter->book_id); + $this->assertCount(1, $chapter->tags); + $firstChapterTag = $chapter->tags()->first(); + $this->assertEquals('Reviewed', $firstChapterTag->name); + $this->assertEquals('2024', $firstChapterTag->value); + $this->assertCount(2, $chapter->pages); + + // Page checks + /** @var Page $pageA */ + $pageA = $chapter->pages->first(); + $this->assertEquals('Page A', $pageA->name); + $this->assertCount(1, $pageA->tags); + $this->assertCount(1, $pageA->attachments); + $pageAImages = Image::where('uploaded_to', '=', $pageA->id)->whereIn('type', ['gallery', 'drawio'])->get(); + $this->assertCount(1, $pageAImages); + + // Reference checks + $attachment = $pageA->attachments->first(); + $this->assertStringContainsString($pageA->getUrl(), $chapter->description_html); + $this->assertStringContainsString($chapter->getUrl(), $pageA->html); + $this->assertStringContainsString($pageAImages[0]->url, $pageA->html); + $this->assertStringContainsString($attachment->getUrl(), $pageA->html); + + ZipTestHelper::deleteZipForImport($import); + } + + public function test_page_import() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $testFilePath = $this->files->testFilePath('test-file.txt'); + $parent = $this->entities->chapter(); + + $import = ZipTestHelper::importFromData([], [ + 'page' => [ + 'id' => 3, + 'name' => 'Page A', + 'priority' => 6, + 'html' => '

    Link to self

    +

    Link to dog drawing

    +

    Link to text attachment

    ', + 'tags' => [ + ['name' => 'Unreviewed'], + ], + 'attachments' => [ + [ + 'id' => 4, + 'name' => 'Text attachment', + 'file' => 'file_attachment' + ] + ], + 'images' => [ + [ + 'id' => 2, + 'name' => 'Dog Drawing', + 'type' => 'drawio', + 'file' => 'dog_image' + ] + ], + ], + ], [ + 'file_attachment' => $testFilePath, + 'dog_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Page $page */ + $page = $this->runner->run($import, $parent); + + // Page checks + $this->assertEquals('Page A', $page->name); + $this->assertCount(1, $page->tags); + $this->assertCount(1, $page->attachments); + $pageImages = Image::where('uploaded_to', '=', $page->id)->whereIn('type', ['gallery', 'drawio'])->get(); + $this->assertCount(1, $pageImages); + $this->assertFileEquals($testImagePath, public_path($pageImages[0]->path)); + + // Reference checks + $this->assertStringContainsString($page->getUrl(), $page->html); + $this->assertStringContainsString($pageImages[0]->url, $page->html); + $this->assertStringContainsString($page->attachments->first()->getUrl(), $page->html); + + ZipTestHelper::deleteZipForImport($import); + } + + public function test_revert_cleans_up_uploaded_files() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $testFilePath = $this->files->testFilePath('test-file.txt'); + $parent = $this->entities->chapter(); + + $import = ZipTestHelper::importFromData([], [ + 'page' => [ + 'name' => 'Page A', + 'html' => '

    Hello

    ', + 'attachments' => [ + [ + 'name' => 'Text attachment', + 'file' => 'file_attachment' + ] + ], + 'images' => [ + [ + 'name' => 'Dog Image', + 'type' => 'gallery', + 'file' => 'dog_image' + ] + ], + ], + ], [ + 'file_attachment' => $testFilePath, + 'dog_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Page $page */ + $page = $this->runner->run($import, $parent); + + $attachment = $page->attachments->first(); + $image = Image::query()->where('uploaded_to', '=', $page->id)->where('type', '=', 'gallery')->first(); + + $this->assertFileExists(public_path($image->path)); + $this->assertFileExists(storage_path($attachment->path)); + + $this->runner->revertStoredFiles(); + + $this->assertFileDoesNotExist(public_path($image->path)); + $this->assertFileDoesNotExist(storage_path($attachment->path)); + + ZipTestHelper::deleteZipForImport($import); + } } From e2f6e50df4347579e3b6eb8e7c48bfcb79199a64 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 18 Nov 2024 15:53:21 +0000 Subject: [PATCH 33/41] ZIP Exports: Added ID checks and testing to validator --- .../ZipExports/Models/ZipExportAttachment.php | 2 +- .../ZipExports/Models/ZipExportBook.php | 2 +- .../ZipExports/Models/ZipExportChapter.php | 2 +- .../ZipExports/Models/ZipExportImage.php | 2 +- .../ZipExports/Models/ZipExportPage.php | 2 +- .../ZipExports/ZipFileReferenceRule.php | 1 - app/Exports/ZipExports/ZipUniqueIdRule.php | 26 +++++++ .../ZipExports/ZipValidationHelper.php | 24 ++++++ lang/en/validation.php | 1 + tests/Exports/ZipExportValidatorTests.php | 74 +++++++++++++++++++ 10 files changed, 130 insertions(+), 6 deletions(-) create mode 100644 app/Exports/ZipExports/ZipUniqueIdRule.php create mode 100644 tests/Exports/ZipExportValidatorTests.php diff --git a/app/Exports/ZipExports/Models/ZipExportAttachment.php b/app/Exports/ZipExports/Models/ZipExportAttachment.php index c6615e1dc..4f5b2f236 100644 --- a/app/Exports/ZipExports/Models/ZipExportAttachment.php +++ b/app/Exports/ZipExports/Models/ZipExportAttachment.php @@ -43,7 +43,7 @@ class ZipExportAttachment extends ZipExportModel public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('attachment')], 'name' => ['required', 'string', 'min:1'], 'link' => ['required_without:file', 'nullable', 'string'], 'file' => ['required_without:link', 'nullable', 'string', $context->fileReferenceRule()], diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php index 0dc4e93d4..47ab8f0a6 100644 --- a/app/Exports/ZipExports/Models/ZipExportBook.php +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -70,7 +70,7 @@ class ZipExportBook extends ZipExportModel public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('book')], 'name' => ['required', 'string', 'min:1'], 'description_html' => ['nullable', 'string'], 'cover' => ['nullable', 'string', $context->fileReferenceRule()], diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php index 50440d61a..5a5fe350f 100644 --- a/app/Exports/ZipExports/Models/ZipExportChapter.php +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -59,7 +59,7 @@ class ZipExportChapter extends ZipExportModel public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('chapter')], 'name' => ['required', 'string', 'min:1'], 'description_html' => ['nullable', 'string'], 'priority' => ['nullable', 'int'], diff --git a/app/Exports/ZipExports/Models/ZipExportImage.php b/app/Exports/ZipExports/Models/ZipExportImage.php index 691eb918f..89083b15b 100644 --- a/app/Exports/ZipExports/Models/ZipExportImage.php +++ b/app/Exports/ZipExports/Models/ZipExportImage.php @@ -33,7 +33,7 @@ class ZipExportImage extends ZipExportModel public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('image')], 'name' => ['required', 'string', 'min:1'], 'file' => ['required', 'string', $context->fileReferenceRule()], 'type' => ['required', 'string', Rule::in(['gallery', 'drawio'])], diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index 3a876e7aa..16e7e9255 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -68,7 +68,7 @@ class ZipExportPage extends ZipExportModel public static function validate(ZipValidationHelper $context, array $data): array { $rules = [ - 'id' => ['nullable', 'int'], + 'id' => ['nullable', 'int', $context->uniqueIdRule('page')], 'name' => ['required', 'string', 'min:1'], 'html' => ['nullable', 'string'], 'markdown' => ['nullable', 'string'], diff --git a/app/Exports/ZipExports/ZipFileReferenceRule.php b/app/Exports/ZipExports/ZipFileReferenceRule.php index bcd3c39ac..7d6c829cf 100644 --- a/app/Exports/ZipExports/ZipFileReferenceRule.php +++ b/app/Exports/ZipExports/ZipFileReferenceRule.php @@ -4,7 +4,6 @@ namespace BookStack\Exports\ZipExports; use Closure; use Illuminate\Contracts\Validation\ValidationRule; -use ZipArchive; class ZipFileReferenceRule implements ValidationRule { diff --git a/app/Exports/ZipExports/ZipUniqueIdRule.php b/app/Exports/ZipExports/ZipUniqueIdRule.php new file mode 100644 index 000000000..ea2b25392 --- /dev/null +++ b/app/Exports/ZipExports/ZipUniqueIdRule.php @@ -0,0 +1,26 @@ +context->hasIdBeenUsed($this->modelType, $value)) { + $fail('validation.zip_unique')->translate(['attribute' => $attribute]); + } + } +} diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php index 55c86b03b..7659c228b 100644 --- a/app/Exports/ZipExports/ZipValidationHelper.php +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -9,6 +9,13 @@ class ZipValidationHelper { protected Factory $validationFactory; + /** + * Local store of validated IDs (in format ":". Example: "book:2") + * which we can use to check uniqueness. + * @var array + */ + protected array $validatedIds = []; + public function __construct( public ZipExportReader $zipReader, ) { @@ -31,6 +38,23 @@ class ZipValidationHelper return new ZipFileReferenceRule($this); } + public function uniqueIdRule(string $type): ZipUniqueIdRule + { + return new ZipUniqueIdRule($this, $type); + } + + public function hasIdBeenUsed(string $type, int $id): bool + { + $key = $type . ':' . $id; + if (isset($this->validatedIds[$key])) { + return true; + } + + $this->validatedIds[$key] = true; + + return false; + } + /** * Validate an array of relation data arrays that are expected * to be for the given ZipExportModel. diff --git a/lang/en/validation.php b/lang/en/validation.php index bc01ac47b..fdfc3d9a9 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -107,6 +107,7 @@ return [ 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', 'zip_model_expected' => 'Data object expected but ":type" found.', + 'zip_unique' => 'The :attribute must be unique for the object type within the ZIP.', // Custom validation lines 'custom' => [ diff --git a/tests/Exports/ZipExportValidatorTests.php b/tests/Exports/ZipExportValidatorTests.php new file mode 100644 index 000000000..4cacea95e --- /dev/null +++ b/tests/Exports/ZipExportValidatorTests.php @@ -0,0 +1,74 @@ +filesToRemove as $file) { + unlink($file); + } + + parent::tearDown(); + } + + protected function getValidatorForData(array $zipData, array $files = []): ZipExportValidator + { + $upload = ZipTestHelper::zipUploadFromData($zipData, $files); + $path = $upload->getRealPath(); + $this->filesToRemove[] = $path; + $reader = new ZipExportReader($path); + return new ZipExportValidator($reader); + } + + public function test_ids_have_to_be_unique() + { + $validator = $this->getValidatorForData([ + 'book' => [ + 'id' => 4, + 'name' => 'My book', + 'pages' => [ + [ + 'id' => 4, + 'name' => 'My page', + 'markdown' => 'hello', + 'attachments' => [ + ['id' => 4, 'name' => 'Attachment A', 'link' => 'https://example.com'], + ['id' => 4, 'name' => 'Attachment B', 'link' => 'https://example.com'] + ], + 'images' => [ + ['id' => 4, 'name' => 'Image A', 'type' => 'gallery', 'file' => 'cat'], + ['id' => 4, 'name' => 'Image b', 'type' => 'gallery', 'file' => 'cat'], + ], + ], + ['id' => 4, 'name' => 'My page', 'markdown' => 'hello'], + ], + 'chapters' => [ + ['id' => 4, 'name' => 'Chapter 1'], + ['id' => 4, 'name' => 'Chapter 2'] + ] + ] + ], ['cat' => $this->files->testFilePath('test-image.png')]); + + $results = $validator->validate(); + $this->assertCount(4, $results); + + $expectedMessage = 'The id must be unique for the object type within the ZIP.'; + $this->assertEquals($expectedMessage, $results['book.pages.0.attachments.1.id']); + $this->assertEquals($expectedMessage, $results['book.pages.0.images.1.id']); + $this->assertEquals($expectedMessage, $results['book.pages.1.id']); + $this->assertEquals($expectedMessage, $results['book.chapters.1.id']); + } +} From 59cfc087e12c8752b4a9f1760db71a13ad6c121c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 18 Nov 2024 17:42:49 +0000 Subject: [PATCH 34/41] ZIP Imports: Added image type validation/handling Images were missing their extension after import since it was (potentially) not part of the import data. This adds validation via mime sniffing (to match normal image upload checks) and also uses the same logic to sniff out a correct extension. Added tests to cover. Also fixed some existing tests around zip functionality. --- .../ZipExports/Models/ZipExportImage.php | 3 +- app/Exports/ZipExports/ZipExportReader.php | 12 +++++++ .../ZipExports/ZipFileReferenceRule.php | 12 +++++++ app/Exports/ZipExports/ZipImportRunner.php | 8 ++++- .../ZipExports/ZipValidationHelper.php | 6 ++-- lang/en/validation.php | 1 + tests/Exports/ZipExportTest.php | 4 --- ...orTests.php => ZipExportValidatorTest.php} | 21 ++++++++++- tests/Exports/ZipImportRunnerTest.php | 35 +++++++++++++++++++ 9 files changed, 92 insertions(+), 10 deletions(-) rename tests/Exports/{ZipExportValidatorTests.php => ZipExportValidatorTest.php} (77%) diff --git a/app/Exports/ZipExports/Models/ZipExportImage.php b/app/Exports/ZipExports/Models/ZipExportImage.php index 89083b15b..e0e7d1198 100644 --- a/app/Exports/ZipExports/Models/ZipExportImage.php +++ b/app/Exports/ZipExports/Models/ZipExportImage.php @@ -32,10 +32,11 @@ class ZipExportImage extends ZipExportModel public static function validate(ZipValidationHelper $context, array $data): array { + $acceptedImageTypes = ['image/png', 'image/jpeg', 'image/gif', 'image/webp']; $rules = [ 'id' => ['nullable', 'int', $context->uniqueIdRule('image')], 'name' => ['required', 'string', 'min:1'], - 'file' => ['required', 'string', $context->fileReferenceRule()], + 'file' => ['required', 'string', $context->fileReferenceRule($acceptedImageTypes)], 'type' => ['required', 'string', Rule::in(['gallery', 'drawio'])], ]; diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php index ebc2fbbc9..6b88ef61c 100644 --- a/app/Exports/ZipExports/ZipExportReader.php +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -7,6 +7,7 @@ use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; use BookStack\Exports\ZipExports\Models\ZipExportModel; use BookStack\Exports\ZipExports\Models\ZipExportPage; +use BookStack\Util\WebSafeMimeSniffer; use ZipArchive; class ZipExportReader @@ -81,6 +82,17 @@ class ZipExportReader return $this->zip->getStream("files/{$fileName}"); } + /** + * Sniff the mime type from the file of given name. + */ + public function sniffFileMime(string $fileName): string + { + $stream = $this->streamFile($fileName); + $sniffContent = fread($stream, 2000); + + return (new WebSafeMimeSniffer())->sniff($sniffContent); + } + /** * @throws ZipExportException */ diff --git a/app/Exports/ZipExports/ZipFileReferenceRule.php b/app/Exports/ZipExports/ZipFileReferenceRule.php index 7d6c829cf..90e78c060 100644 --- a/app/Exports/ZipExports/ZipFileReferenceRule.php +++ b/app/Exports/ZipExports/ZipFileReferenceRule.php @@ -9,6 +9,7 @@ class ZipFileReferenceRule implements ValidationRule { public function __construct( protected ZipValidationHelper $context, + protected array $acceptedMimes, ) { } @@ -21,5 +22,16 @@ class ZipFileReferenceRule implements ValidationRule if (!$this->context->zipReader->fileExists($value)) { $fail('validation.zip_file')->translate(); } + + if (!empty($this->acceptedMimes)) { + $fileMime = $this->context->zipReader->sniffFileMime($value); + if (!in_array($fileMime, $this->acceptedMimes)) { + $fail('validation.zip_file_mime')->translate([ + 'attribute' => $attribute, + 'validTypes' => implode(',', $this->acceptedMimes), + 'foundType' => $fileMime + ]); + } + } } } diff --git a/app/Exports/ZipExports/ZipImportRunner.php b/app/Exports/ZipExports/ZipImportRunner.php index 27d859e59..d25a1621f 100644 --- a/app/Exports/ZipExports/ZipImportRunner.php +++ b/app/Exports/ZipExports/ZipImportRunner.php @@ -228,6 +228,9 @@ class ZipImportRunner protected function importImage(ZipExportImage $exportImage, Page $page, ZipExportReader $reader): Image { + $mime = $reader->sniffFileMime($exportImage->file); + $extension = explode('/', $mime)[1]; + $file = $this->zipFileToUploadedFile($exportImage->file, $reader); $image = $this->imageService->saveNewFromUpload( $file, @@ -236,9 +239,12 @@ class ZipImportRunner null, null, true, - $exportImage->name, + $exportImage->name . '.' . $extension, ); + $image->name = $exportImage->name; + $image->save(); + $this->references->addImage($image, $exportImage->id); return $image; diff --git a/app/Exports/ZipExports/ZipValidationHelper.php b/app/Exports/ZipExports/ZipValidationHelper.php index 7659c228b..fd9cd7844 100644 --- a/app/Exports/ZipExports/ZipValidationHelper.php +++ b/app/Exports/ZipExports/ZipValidationHelper.php @@ -33,9 +33,9 @@ class ZipValidationHelper return $messages; } - public function fileReferenceRule(): ZipFileReferenceRule + public function fileReferenceRule(array $acceptedMimes = []): ZipFileReferenceRule { - return new ZipFileReferenceRule($this); + return new ZipFileReferenceRule($this, $acceptedMimes); } public function uniqueIdRule(string $type): ZipUniqueIdRule @@ -43,7 +43,7 @@ class ZipValidationHelper return new ZipUniqueIdRule($this, $type); } - public function hasIdBeenUsed(string $type, int $id): bool + public function hasIdBeenUsed(string $type, mixed $id): bool { $key = $type . ':' . $id; if (isset($this->validatedIds[$key])) { diff --git a/lang/en/validation.php b/lang/en/validation.php index fdfc3d9a9..d9b982d1e 100644 --- a/lang/en/validation.php +++ b/lang/en/validation.php @@ -106,6 +106,7 @@ return [ 'uploaded' => 'The file could not be uploaded. The server may not accept files of this size.', 'zip_file' => 'The :attribute needs to reference a file within the ZIP.', + 'zip_file_mime' => 'The :attribute needs to reference a file of type :validTypes, found :foundType.', 'zip_model_expected' => 'Data object expected but ":type" found.', 'zip_unique' => 'The :attribute must be unique for the object type within the ZIP.', diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index ac07b33ae..12531239f 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -107,12 +107,10 @@ class ZipExportTest extends TestCase [ 'name' => 'Exporty', 'value' => 'Content', - 'order' => 1, ], [ 'name' => 'Another', 'value' => '', - 'order' => 2, ] ], $pageData['tags']); } @@ -162,7 +160,6 @@ class ZipExportTest extends TestCase $attachmentData = $pageData['attachments'][0]; $this->assertEquals('PageAttachmentExport.txt', $attachmentData['name']); $this->assertEquals($attachment->id, $attachmentData['id']); - $this->assertEquals(1, $attachmentData['order']); $this->assertArrayNotHasKey('link', $attachmentData); $this->assertNotEmpty($attachmentData['file']); @@ -193,7 +190,6 @@ class ZipExportTest extends TestCase $attachmentData = $pageData['attachments'][0]; $this->assertEquals('My link attachment for export', $attachmentData['name']); $this->assertEquals($attachment->id, $attachmentData['id']); - $this->assertEquals(1, $attachmentData['order']); $this->assertEquals('https://example.com/cats', $attachmentData['link']); $this->assertArrayNotHasKey('file', $attachmentData); } diff --git a/tests/Exports/ZipExportValidatorTests.php b/tests/Exports/ZipExportValidatorTest.php similarity index 77% rename from tests/Exports/ZipExportValidatorTests.php rename to tests/Exports/ZipExportValidatorTest.php index 4cacea95e..c453ef294 100644 --- a/tests/Exports/ZipExportValidatorTests.php +++ b/tests/Exports/ZipExportValidatorTest.php @@ -11,7 +11,7 @@ use BookStack\Exports\ZipExports\ZipImportRunner; use BookStack\Uploads\Image; use Tests\TestCase; -class ZipExportValidatorTests extends TestCase +class ZipExportValidatorTest extends TestCase { protected array $filesToRemove = []; @@ -71,4 +71,23 @@ class ZipExportValidatorTests extends TestCase $this->assertEquals($expectedMessage, $results['book.pages.1.id']); $this->assertEquals($expectedMessage, $results['book.chapters.1.id']); } + + public function test_image_files_need_to_be_a_valid_detected_image_file() + { + $validator = $this->getValidatorForData([ + 'page' => [ + 'id' => 4, + 'name' => 'My page', + 'markdown' => 'hello', + 'images' => [ + ['id' => 4, 'name' => 'Image A', 'type' => 'gallery', 'file' => 'cat'], + ], + ] + ], ['cat' => $this->files->testFilePath('test-file.txt')]); + + $results = $validator->validate(); + $this->assertCount(1, $results); + + $this->assertEquals('The file needs to reference a file of type image/png,image/jpeg,image/gif,image/webp, found text/plain.', $results['page.images.0.file']); + } } diff --git a/tests/Exports/ZipImportRunnerTest.php b/tests/Exports/ZipImportRunnerTest.php index c833fadda..d3af6df76 100644 --- a/tests/Exports/ZipImportRunnerTest.php +++ b/tests/Exports/ZipImportRunnerTest.php @@ -358,4 +358,39 @@ class ZipImportRunnerTest extends TestCase ZipTestHelper::deleteZipForImport($import); } + + public function test_imported_images_have_their_detected_extension_added() + { + $testImagePath = $this->files->testFilePath('test-image.png'); + $parent = $this->entities->chapter(); + + $import = ZipTestHelper::importFromData([], [ + 'page' => [ + 'name' => 'Page A', + 'html' => '

    hello

    ', + 'images' => [ + [ + 'id' => 2, + 'name' => 'Cat', + 'type' => 'gallery', + 'file' => 'cat_image' + ] + ], + ], + ], [ + 'cat_image' => $testImagePath, + ]); + + $this->asAdmin(); + /** @var Page $page */ + $page = $this->runner->run($import, $parent); + + $pageImages = Image::where('uploaded_to', '=', $page->id)->whereIn('type', ['gallery', 'drawio'])->get(); + + $this->assertCount(1, $pageImages); + $this->assertStringEndsWith('.png', $pageImages[0]->url); + $this->assertStringEndsWith('.png', $pageImages[0]->path); + + ZipTestHelper::deleteZipForImport($import); + } } From c0dff6d4a6227549e7f756cdd0d7cd6a003b9886 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Fri, 22 Nov 2024 21:03:04 +0000 Subject: [PATCH 35/41] ZIP Imports: Added book content ordering to import preview --- app/Exports/ZipExports/Models/ZipExportBook.php | 14 ++++++++++++++ app/Exports/ZipExports/Models/ZipExportChapter.php | 7 ++++++- app/Exports/ZipExports/Models/ZipExportPage.php | 2 +- app/Exports/ZipExports/ZipExportReader.php | 1 - .../views/exports/parts/import-item.blade.php | 14 ++++++++------ 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/app/Exports/ZipExports/Models/ZipExportBook.php b/app/Exports/ZipExports/Models/ZipExportBook.php index 47ab8f0a6..4f641d25b 100644 --- a/app/Exports/ZipExports/Models/ZipExportBook.php +++ b/app/Exports/ZipExports/Models/ZipExportBook.php @@ -36,6 +36,20 @@ class ZipExportBook extends ZipExportModel } } + public function children(): array + { + $children = [ + ...$this->pages, + ...$this->chapters, + ]; + + usort($children, function ($a, $b) { + return ($a->priority ?? 0) - ($b->priority ?? 0); + }); + + return $children; + } + public static function fromModel(Book $model, ZipExportFiles $files): self { $instance = new self(); diff --git a/app/Exports/ZipExports/Models/ZipExportChapter.php b/app/Exports/ZipExports/Models/ZipExportChapter.php index 5a5fe350f..bf2dc78f8 100644 --- a/app/Exports/ZipExports/Models/ZipExportChapter.php +++ b/app/Exports/ZipExports/Models/ZipExportChapter.php @@ -20,7 +20,7 @@ class ZipExportChapter extends ZipExportModel public function metadataOnly(): void { - $this->description_html = $this->priority = null; + $this->description_html = null; foreach ($this->pages as $page) { $page->metadataOnly(); @@ -30,6 +30,11 @@ class ZipExportChapter extends ZipExportModel } } + public function children(): array + { + return $this->pages; + } + public static function fromModel(Chapter $model, ZipExportFiles $files): self { $instance = new self(); diff --git a/app/Exports/ZipExports/Models/ZipExportPage.php b/app/Exports/ZipExports/Models/ZipExportPage.php index 16e7e9255..097443df0 100644 --- a/app/Exports/ZipExports/Models/ZipExportPage.php +++ b/app/Exports/ZipExports/Models/ZipExportPage.php @@ -23,7 +23,7 @@ class ZipExportPage extends ZipExportModel public function metadataOnly(): void { - $this->html = $this->markdown = $this->priority = null; + $this->html = $this->markdown = null; foreach ($this->attachments as $attachment) { $attachment->metadataOnly(); diff --git a/app/Exports/ZipExports/ZipExportReader.php b/app/Exports/ZipExports/ZipExportReader.php index 6b88ef61c..c3d5c23cf 100644 --- a/app/Exports/ZipExports/ZipExportReader.php +++ b/app/Exports/ZipExports/ZipExportReader.php @@ -5,7 +5,6 @@ namespace BookStack\Exports\ZipExports; use BookStack\Exceptions\ZipExportException; use BookStack\Exports\ZipExports\Models\ZipExportBook; use BookStack\Exports\ZipExports\Models\ZipExportChapter; -use BookStack\Exports\ZipExports\Models\ZipExportModel; use BookStack\Exports\ZipExports\Models\ZipExportPage; use BookStack\Util\WebSafeMimeSniffer; use ZipArchive; diff --git a/resources/views/exports/parts/import-item.blade.php b/resources/views/exports/parts/import-item.blade.php index 811a3b31b..5da4b2140 100644 --- a/resources/views/exports/parts/import-item.blade.php +++ b/resources/views/exports/parts/import-item.blade.php @@ -16,11 +16,13 @@ $model - object @icon('tag'){{ count($model->tags) }} @endif
    - @foreach($model->chapters ?? [] as $chapter) - @include('exports.parts.import-item', ['type' => 'chapter', 'model' => $chapter]) - @endforeach - @foreach($model->pages ?? [] as $page) - @include('exports.parts.import-item', ['type' => 'page', 'model' => $page]) - @endforeach + @if(method_exists($model, 'children')) + @foreach($model->children() as $child) + @include('exports.parts.import-item', [ + 'type' => ($child instanceof \BookStack\Exports\ZipExports\Models\ZipExportPage) ? 'page' : 'chapter', + 'model' => $child + ]) + @endforeach + @endif
    \ No newline at end of file From f79c6aef8d1c9aa83e9ce89ec1e5ac9d9e1eb570 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Fri, 22 Nov 2024 21:36:42 +0000 Subject: [PATCH 36/41] ZIP Imports: Updated import form to show loading indicator And disable button after submit. Added here because the import could take some time, so it's best to show an indicator to the user to show that something is happening, and help prevent duplicate submission or re-submit attempts. --- resources/js/components/index.js | 1 + resources/js/components/loading-button.ts | 38 +++++++++++++++++++ resources/sass/styles.scss | 4 ++ resources/views/exports/import-show.blade.php | 4 +- 4 files changed, 45 insertions(+), 2 deletions(-) create mode 100644 resources/js/components/loading-button.ts diff --git a/resources/js/components/index.js b/resources/js/components/index.js index 8ad5e14cb..12c991a51 100644 --- a/resources/js/components/index.js +++ b/resources/js/components/index.js @@ -30,6 +30,7 @@ export {HeaderMobileToggle} from './header-mobile-toggle'; export {ImageManager} from './image-manager'; export {ImagePicker} from './image-picker'; export {ListSortControl} from './list-sort-control'; +export {LoadingButton} from './loading-button'; export {MarkdownEditor} from './markdown-editor'; export {NewUserPassword} from './new-user-password'; export {Notification} from './notification'; diff --git a/resources/js/components/loading-button.ts b/resources/js/components/loading-button.ts new file mode 100644 index 000000000..a793d30a2 --- /dev/null +++ b/resources/js/components/loading-button.ts @@ -0,0 +1,38 @@ +import {Component} from "./component.js"; +import {showLoading} from "../services/dom"; +import {el} from "../wysiwyg/utils/dom"; + +/** + * Loading button. + * Shows a loading indicator and disables the button when the button is clicked, + * or when the form attached to the button is submitted. + */ +export class LoadingButton extends Component { + + protected button!: HTMLButtonElement; + protected loadingEl: HTMLDivElement|null = null; + + setup() { + this.button = this.$el as HTMLButtonElement; + const form = this.button.form; + + const action = () => { + setTimeout(() => this.showLoadingState(), 10) + }; + + this.button.addEventListener('click', action); + if (form) { + form.addEventListener('submit', action); + } + } + + showLoadingState() { + this.button.disabled = true; + + if (!this.loadingEl) { + this.loadingEl = el('div', {class: 'inline block'}) as HTMLDivElement; + showLoading(this.loadingEl); + this.button.after(this.loadingEl); + } + } +} \ No newline at end of file diff --git a/resources/sass/styles.scss b/resources/sass/styles.scss index 2cf3cbf82..2106f86e6 100644 --- a/resources/sass/styles.scss +++ b/resources/sass/styles.scss @@ -106,6 +106,10 @@ $loadingSize: 10px; } } +.inline.block .loading-container { + margin: $-xs $-s; +} + .skip-to-content-link { position: fixed; top: -52px; diff --git a/resources/views/exports/import-show.blade.php b/resources/views/exports/import-show.blade.php index e4f199aa2..a28b79bb3 100644 --- a/resources/views/exports/import-show.blade.php +++ b/resources/views/exports/import-show.blade.php @@ -59,7 +59,7 @@ ]) @endif -
    +
    {{ trans('common.cancel') }}
    - +
    From 9ecc91929a60a66ff7e821dfbc9d2b55197988f7 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 25 Nov 2024 15:54:15 +0000 Subject: [PATCH 37/41] ZIP Import & Exports: Addressed issues during testing - Handled links to within-zip page images found in chapter/book descriptions; Added test to cover. - Fixed session showing unrelated success on failed import. Tested import file-create undo on failure as part of this testing. --- app/Exports/Controllers/ImportController.php | 2 ++ .../ZipExports/ZipExportReferences.php | 7 ++--- lang/en/errors.php | 1 + tests/Exports/ZipExportTest.php | 26 +++++++++++++++++++ 4 files changed, 33 insertions(+), 3 deletions(-) diff --git a/app/Exports/Controllers/ImportController.php b/app/Exports/Controllers/ImportController.php index a20c341fb..b938dac8e 100644 --- a/app/Exports/Controllers/ImportController.php +++ b/app/Exports/Controllers/ImportController.php @@ -89,6 +89,8 @@ class ImportController extends Controller try { $entity = $this->imports->runImport($import, $parent); } catch (ZipImportException $exception) { + session()->flush(); + $this->showErrorNotification(trans('errors.import_zip_failed_notification')); return redirect($import->getUrl())->with('import_errors', $exception->errors); } diff --git a/app/Exports/ZipExports/ZipExportReferences.php b/app/Exports/ZipExports/ZipExportReferences.php index 0de409fa1..bf5e02133 100644 --- a/app/Exports/ZipExports/ZipExportReferences.php +++ b/app/Exports/ZipExports/ZipExportReferences.php @@ -127,11 +127,12 @@ class ZipExportReferences return null; } - // We don't expect images to be part of book/chapter content - if (!($exportModel instanceof ZipExportPage)) { - return null; + // Handle simple links outside of page content + if (!($exportModel instanceof ZipExportPage) && isset($this->images[$model->id])) { + return "[[bsexport:image:{$model->id}]]"; } + // Find and include images if in visibility $page = $model->getPage(); if ($page && userCan('view', $page)) { if (!isset($this->images[$model->id])) { diff --git a/lang/en/errors.php b/lang/en/errors.php index ced80a32c..9d7383796 100644 --- a/lang/en/errors.php +++ b/lang/en/errors.php @@ -110,6 +110,7 @@ return [ 'import_zip_cant_decode_data' => 'Could not find and decode ZIP data.json content.', 'import_zip_no_data' => 'ZIP file data has no expected book, chapter or page content.', 'import_validation_failed' => 'Import ZIP failed to validate with errors:', + 'import_zip_failed_notification' => 'Failed to import ZIP file.', 'import_perms_books' => 'You are lacking the required permissions to create books.', 'import_perms_chapters' => 'You are lacking the required permissions to create chapters.', 'import_perms_pages' => 'You are lacking the required permissions to create pages.', diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index 12531239f..6e8462f59 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -274,6 +274,32 @@ class ZipExportTest extends TestCase $this->assertStringContainsString('href="[[bsexport:book:' . $book->id . ']]?view=true"', $pageData['html']); } + public function test_book_and_chapter_description_links_to_images_in_pages_are_converted() + { + $book = $this->entities->bookHasChaptersAndPages(); + $chapter = $book->chapters()->first(); + $page = $chapter->pages()->first(); + + $this->asEditor(); + $this->files->uploadGalleryImageToPage($this, $page); + /** @var Image $image */ + $image = Image::query()->where('type', '=', 'gallery') + ->where('uploaded_to', '=', $page->id)->first(); + + $book->description_html = '

    Link to image

    '; + $book->save(); + $chapter->description_html = '

    Link to image

    '; + $chapter->save(); + + $zipResp = $this->get($book->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $bookData = $zip->data['book']; + $chapterData = $bookData['chapters'][0]; + + $this->assertStringContainsString('href="[[bsexport:image:' . $image->id . ']]"', $bookData['description_html']); + $this->assertStringContainsString('href="[[bsexport:image:' . $image->id . ']]"', $chapterData['description_html']); + } + public function test_cross_reference_links_external_to_export_are_not_converted() { $page = $this->entities->page(); From 95d62e7f573b5bbb04a66fae926c8a33c6ab5c43 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Mon, 25 Nov 2024 16:23:59 +0000 Subject: [PATCH 38/41] ZIP Imports/Exports: Fixed some lint and test issues - Updated test handling to create imports folder when required. - Updated some tests to delete created import zip files. --- resources/js/components/index.js | 2 +- resources/js/components/page-comments.js | 1 - tests/Exports/ZipImportTest.php | 8 ++++++++ tests/Exports/ZipTestHelper.php | 9 ++++++++- 4 files changed, 17 insertions(+), 3 deletions(-) diff --git a/resources/js/components/index.js b/resources/js/components/index.js index 12c991a51..24e60bd97 100644 --- a/resources/js/components/index.js +++ b/resources/js/components/index.js @@ -30,7 +30,7 @@ export {HeaderMobileToggle} from './header-mobile-toggle'; export {ImageManager} from './image-manager'; export {ImagePicker} from './image-picker'; export {ListSortControl} from './list-sort-control'; -export {LoadingButton} from './loading-button'; +export {LoadingButton} from './loading-button.ts'; export {MarkdownEditor} from './markdown-editor'; export {NewUserPassword} from './new-user-password'; export {Notification} from './notification'; diff --git a/resources/js/components/page-comments.js b/resources/js/components/page-comments.js index 1d6abfe20..63900888a 100644 --- a/resources/js/components/page-comments.js +++ b/resources/js/components/page-comments.js @@ -93,7 +93,6 @@ export class PageComments extends Component { updateCount() { const count = this.getCommentCount(); - console.log('update count', count, this.container); this.commentsTitle.textContent = window.$trans.choice(this.countText, count, {count}); } diff --git a/tests/Exports/ZipImportTest.php b/tests/Exports/ZipImportTest.php index 3644e9bdc..ad0e6b241 100644 --- a/tests/Exports/ZipImportTest.php +++ b/tests/Exports/ZipImportTest.php @@ -168,6 +168,8 @@ class ZipImportTest extends TestCase $resp->assertRedirect("/import/{$import->id}"); $this->assertFileExists(storage_path($import->path)); $this->assertActivityExists(ActivityType::IMPORT_CREATE); + + ZipTestHelper::deleteZipForImport($import); } public function test_import_show_page() @@ -325,6 +327,8 @@ class ZipImportTest extends TestCase $resp = $this->followRedirects($resp); $resp->assertSeeText('The name field is required.'); $resp->assertSeeText('The id must be an integer.'); + + ZipTestHelper::deleteZipForImport($import); } public function test_run_checks_permissions_on_import() @@ -340,6 +344,8 @@ class ZipImportTest extends TestCase $resp = $this->followRedirects($resp); $resp->assertSeeText('You are lacking the required permissions to create books.'); + + ZipTestHelper::deleteZipForImport($import); } public function test_run_requires_parent_for_chapter_and_page_imports() @@ -379,6 +385,8 @@ class ZipImportTest extends TestCase $resp = $this->followRedirects($resp); $resp->assertSee('Parent book required for chapter import.'); + + ZipTestHelper::deleteZipForImport($import); } protected function runImportFromFile(UploadedFile $file): TestResponse diff --git a/tests/Exports/ZipTestHelper.php b/tests/Exports/ZipTestHelper.php index 2196f361c..d830d8eb6 100644 --- a/tests/Exports/ZipTestHelper.php +++ b/tests/Exports/ZipTestHelper.php @@ -20,7 +20,14 @@ class ZipTestHelper $import = Import::factory()->create($importData); $zip = static::zipUploadFromData($zipData, $files); - rename($zip->getRealPath(), storage_path($import->path)); + $targetPath = storage_path($import->path); + $targetDir = dirname($targetPath); + + if (!file_exists($targetDir)) { + mkdir($targetDir); + } + + rename($zip->getRealPath(), $targetPath); return $import; } From 0a182a45ba944c807c7a5ba6d6ba3a48809e1dc2 Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 26 Nov 2024 15:59:39 +0000 Subject: [PATCH 39/41] ZIP Exports: Added detection/handling of images with external storage Added test to cover. --- app/Exports/ZipExports/ZipReferenceParser.php | 23 +++++++++++++-- .../ModelResolvers/ImageModelResolver.php | 29 +++++++++++++++++-- app/Uploads/ImageStorage.php | 16 ++++++++-- tests/Exports/ZipExportTest.php | 24 +++++++++++++++ 4 files changed, 85 insertions(+), 7 deletions(-) diff --git a/app/Exports/ZipExports/ZipReferenceParser.php b/app/Exports/ZipExports/ZipReferenceParser.php index 5929383b4..a6560e3f2 100644 --- a/app/Exports/ZipExports/ZipReferenceParser.php +++ b/app/Exports/ZipExports/ZipReferenceParser.php @@ -11,6 +11,7 @@ use BookStack\References\ModelResolvers\CrossLinkModelResolver; use BookStack\References\ModelResolvers\ImageModelResolver; use BookStack\References\ModelResolvers\PageLinkModelResolver; use BookStack\References\ModelResolvers\PagePermalinkModelResolver; +use BookStack\Uploads\ImageStorage; class ZipReferenceParser { @@ -33,8 +34,7 @@ class ZipReferenceParser */ public function parseLinks(string $content, callable $handler): string { - $escapedBase = preg_quote(url('/'), '/'); - $linkRegex = "/({$escapedBase}.*?)[\\t\\n\\f>\"'=?#()]/"; + $linkRegex = $this->getLinkRegex(); $matches = []; preg_match_all($linkRegex, $content, $matches); @@ -118,4 +118,23 @@ class ZipReferenceParser return $this->modelResolvers; } + + /** + * Build the regex to identify links we should handle in content. + */ + protected function getLinkRegex(): string + { + $urls = [rtrim(url('/'), '/')]; + $imageUrl = rtrim(ImageStorage::getPublicUrl('/'), '/'); + if ($urls[0] !== $imageUrl) { + $urls[] = $imageUrl; + } + + + $urlBaseRegex = implode('|', array_map(function ($url) { + return preg_quote($url, '/'); + }, $urls)); + + return "/(({$urlBaseRegex}).*?)[\\t\\n\\f>\"'=?#()]/"; + } } diff --git a/app/References/ModelResolvers/ImageModelResolver.php b/app/References/ModelResolvers/ImageModelResolver.php index 331dd593b..2c6c9fecd 100644 --- a/app/References/ModelResolvers/ImageModelResolver.php +++ b/app/References/ModelResolvers/ImageModelResolver.php @@ -3,19 +3,22 @@ namespace BookStack\References\ModelResolvers; use BookStack\Uploads\Image; +use BookStack\Uploads\ImageStorage; class ImageModelResolver implements CrossLinkModelResolver { + protected ?string $pattern = null; + public function resolve(string $link): ?Image { - $pattern = '/^' . preg_quote(url('/uploads/images'), '/') . '\/(.+)/'; + $pattern = $this->getUrlPattern(); $matches = []; $match = preg_match($pattern, $link, $matches); if (!$match) { return null; } - $path = $matches[1]; + $path = $matches[2]; // Strip thumbnail element from path if existing $originalPathSplit = array_filter(explode('/', $path), function (string $part) { @@ -30,4 +33,26 @@ class ImageModelResolver implements CrossLinkModelResolver return Image::query()->where('path', '=', $fullPath)->first(); } + + /** + * Get the regex pattern to identify image URLs. + * Caches the pattern since it requires looking up to settings/config. + */ + protected function getUrlPattern(): string + { + if ($this->pattern) { + return $this->pattern; + } + + $urls = [url('/uploads/images')]; + $baseImageUrl = ImageStorage::getPublicUrl('/uploads/images'); + if ($baseImageUrl !== $urls[0]) { + $urls[] = $baseImageUrl; + } + + $imageUrlRegex = implode('|', array_map(fn ($url) => preg_quote($url, '/'), $urls)); + $this->pattern = '/^(' . $imageUrlRegex . ')\/(.+)/'; + + return $this->pattern; + } } diff --git a/app/Uploads/ImageStorage.php b/app/Uploads/ImageStorage.php index dc4abc0f2..ddaa26a94 100644 --- a/app/Uploads/ImageStorage.php +++ b/app/Uploads/ImageStorage.php @@ -110,10 +110,20 @@ class ImageStorage } /** - * Gets a public facing url for an image by checking relevant environment variables. + * Gets a public facing url for an image or location at the given path. + */ + public static function getPublicUrl(string $filePath): string + { + return static::getPublicBaseUrl() . '/' . ltrim($filePath, '/'); + } + + /** + * Get the public base URL used for images. + * Will not include any path element of the image file, just the base part + * from where the path is then expected to start from. * If s3-style store is in use it will default to guessing a public bucket URL. */ - public function getPublicUrl(string $filePath): string + protected static function getPublicBaseUrl(): string { $storageUrl = config('filesystems.url'); @@ -131,6 +141,6 @@ class ImageStorage $basePath = $storageUrl ?: url('/'); - return rtrim($basePath, '/') . $filePath; + return rtrim($basePath, '/'); } } diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index 6e8462f59..17891c73d 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -300,6 +300,30 @@ class ZipExportTest extends TestCase $this->assertStringContainsString('href="[[bsexport:image:' . $image->id . ']]"', $chapterData['description_html']); } + public function test_image_links_are_handled_when_using_external_storage_url() + { + $page = $this->entities->page(); + + $this->asEditor(); + $this->files->uploadGalleryImageToPage($this, $page); + /** @var Image $image */ + $image = Image::query()->where('type', '=', 'gallery') + ->where('uploaded_to', '=', $page->id)->first(); + + config()->set('filesystems.url', 'https://i.example.com/content'); + + $storageUrl = 'https://i.example.com/content/' . ltrim($image->path, '/'); + $page->html = '

    Original URLStorage URL

    '; + $page->save(); + + $zipResp = $this->get($page->getUrl("/export/zip")); + $zip = $this->extractZipResponse($zipResp); + $pageData = $zip->data['page']; + + $ref = '[[bsexport:image:' . $image->id . ']]'; + $this->assertStringContainsString("Original URLStorage URL", $pageData['html']); + } + public function test_cross_reference_links_external_to_export_are_not_converted() { $page = $this->entities->page(); From edb684c72ce0b1f1cf9be90d338ee08e24b4a0cc Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Tue, 26 Nov 2024 17:53:20 +0000 Subject: [PATCH 40/41] ZIP Exports: Updated format doc with advisories regarding html/md --- dev/docs/portable-zip-file-format.md | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index 7e5df3f01..fbb317858 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -13,7 +13,8 @@ Following the goals & ideals of BookStack, stability is very important. We aim f - Where reasonably possible, we will attempt to avoid modifications/removals of existing features/properties. - Where potentially breaking changes do have to be made, these will be noted in BookStack release/update notes. -The addition of new features/properties alone are not considered as a breaking change to the format. Breaking changes are considered as such where they could impact common/expected use of the existing properties and features we document, they are not considered based upon user assumptions or any possible breakage. For example if your application, using the format, breaks because we added a new property while you hard-coded your application to use the third property (instead of a property name), then that's on you. +The addition of new features/properties alone are not considered as a breaking change to the format. Breaking changes are considered as such where they could impact common/expected use of the existing properties and features we document, they are not considered based upon user assumptions or any possible breakage. +For example if your application, using the format, breaks because we added a new property while you hard-coded your application to use the third property (instead of a property name), then that's on you. ## Format Outline @@ -57,6 +58,23 @@ Here's an example of each type of such reference that could be used: [[bsexport:book:8]] ``` +## HTML & Markdown Content + +BookStack commonly stores & utilises content in the HTML format. +Properties that expect or provided HTML will either be named `html` or contain `html` in the property name. +While BookStack supports a range of HTML, not all HTML content will be supported by BookStack and be assured to work as desired across all BookStack features. +The HTML supported by BookStack is not yet formally documented, but you can inspect to what the WYSIWYG editor produces as a basis. +Generally, top-level elements should keep to common block formats (p, blockquote, h1, h2 etc...) with no nesting or custom structure apart from common inline elements. +Some areas of BookStack where HTML is used, like book & chapter descriptions, will strictly limit/filter HTML tag & attributes to an allow-list. + +For markdown content, in BookStack we target [the commonmark spec](https://commonmark.org/) with the addition of tables & task-lists. +HTML within markdown is supported but not all HTML is assured to work as advised above. + +### Content Security + +If you're consuming HTML or markdown within an export please consider that the content is not assured to be safe, even if provided directly by a BookStack instance. It's best to treat such content as potentially unsafe. +By default, BookStack performs some basic filtering to remove scripts among other potentially dangerous elements but this is not foolproof. BookStack itself relies on additional security mechanisms such as [CSP](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) to help prevent a range of exploits. + ## Export Data - `data.json` The `data.json` file is a JSON format file which contains all structured data for the export. The properties are as follows: @@ -114,9 +132,9 @@ The `pages` are not all pages within the book, just those that are direct childr - `images` - [Image](#image) array, optional, images used in this page. - `tags` - [Tag](#tag) array, optional, tags assigned to this page. -To define the page content, either `markdown` or `html` should be provided. Ideally these should be limited to the range of markdown and HTML which BookStack supports. +To define the page content, either `markdown` or `html` should be provided. Ideally these should be limited to the range of markdown and HTML which BookStack supports. See the ["HTML & Markdown Content"](#html--markdown-content) section. -The page editor type, and edit content will be determined by what content is provided. If non-empty `markdown` is provided, the page will be assumed as a markdown editor page (where permissions allow) and the HTML will be rendered from the markdown content. Otherwise, the provided `html` will be used as editor and display content. +The page editor type, and edit content will be determined by what content is provided. If non-empty `markdown` is provided, the page will be assumed as a markdown editor page (where permissions allow) and the HTML will be rendered from the markdown content. Otherwise, the provided `html` will be used as editor & display content. #### Image From bdca9fc1ce6f3f792106e86348cfb1479f4dd27c Mon Sep 17 00:00:00 2001 From: Dan Brown Date: Wed, 27 Nov 2024 16:30:19 +0000 Subject: [PATCH 41/41] ZIP Exports: Changed the instance id mechanism Adds an instance id via app settings. --- app/Exports/ZipExports/ZipExportBuilder.php | 4 +-- ...4_11_27_171039_add_instance_id_setting.php | 30 +++++++++++++++++++ dev/docs/portable-zip-file-format.md | 6 ++-- tests/Exports/ZipExportTest.php | 6 ++-- 4 files changed, 38 insertions(+), 8 deletions(-) create mode 100644 database/migrations/2024_11_27_171039_add_instance_id_setting.php diff --git a/app/Exports/ZipExports/ZipExportBuilder.php b/app/Exports/ZipExports/ZipExportBuilder.php index 42fb03541..4c5c638f5 100644 --- a/app/Exports/ZipExports/ZipExportBuilder.php +++ b/app/Exports/ZipExports/ZipExportBuilder.php @@ -69,8 +69,8 @@ class ZipExportBuilder $this->data['exported_at'] = date(DATE_ATOM); $this->data['instance'] = [ - 'version' => trim(file_get_contents(base_path('version'))), - 'id_ciphertext' => encrypt('bookstack'), + 'id' => setting('instance-id', ''), + 'version' => trim(file_get_contents(base_path('version'))), ]; $zipFile = tempnam(sys_get_temp_dir(), 'bszip-'); diff --git a/database/migrations/2024_11_27_171039_add_instance_id_setting.php b/database/migrations/2024_11_27_171039_add_instance_id_setting.php new file mode 100644 index 000000000..ee1e90d03 --- /dev/null +++ b/database/migrations/2024_11_27_171039_add_instance_id_setting.php @@ -0,0 +1,30 @@ +insert([ + 'setting_key' => 'instance-id', + 'value' => Str::uuid(), + 'created_at' => Carbon::now(), + 'updated_at' => Carbon::now(), + 'type' => 'string', + ]); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + DB::table('settings')->where('setting_key', '=', 'instance-id')->delete(); + } +}; diff --git a/dev/docs/portable-zip-file-format.md b/dev/docs/portable-zip-file-format.md index fbb317858..754cb4d3e 100644 --- a/dev/docs/portable-zip-file-format.md +++ b/dev/docs/portable-zip-file-format.md @@ -93,12 +93,10 @@ The below details the objects & their properties used in Application Data. #### Instance -These details are mainly informational regarding the exporting BookStack instance from where an export was created from. +These details are informational regarding the exporting BookStack instance from where an export was created from. +- `id` - String, required, unique identifier for the BookStack instance. - `version` - String, required, BookStack version of the export source instance. -- `id_ciphertext` - String, required, identifier for the BookStack instance. - -The `id_ciphertext` is the ciphertext of encrypting the text `bookstack`. This is used as a simple & rough way for a BookStack instance to be able to identify if they were the source (by attempting to decrypt the ciphertext). #### Book diff --git a/tests/Exports/ZipExportTest.php b/tests/Exports/ZipExportTest.php index 17891c73d..ebe07d052 100644 --- a/tests/Exports/ZipExportTest.php +++ b/tests/Exports/ZipExportTest.php @@ -54,8 +54,10 @@ class ZipExportTest extends TestCase $version = trim(file_get_contents(base_path('version'))); $this->assertEquals($version, $zip->data['instance']['version']); - $instanceId = decrypt($zip->data['instance']['id_ciphertext']); - $this->assertEquals('bookstack', $instanceId); + $zipInstanceId = $zip->data['instance']['id']; + $instanceId = setting('instance-id'); + $this->assertNotEmpty($instanceId); + $this->assertEquals($instanceId, $zipInstanceId); } public function test_page_export()