nextcloud

Understanding and improving Nextcloud Previews

NextCloudPi‘s target audience is domestic hosters and one of the use cases that attracts users is to self-host their own picture collection.

The main complaint for such users is that gallery is too slow. When you open the gallery previews are generated on the fly which affects responsiveness. Once they are generated, the next time it will load much faster.

The current solution is to generate the previews offline through a cron job using the Preview Generator App so that they are ready to be fetched when we open the Gallery.

Even when using this App browsing our pictures is usually very slow. This happens typically for the following reasons

  • Lack of computing power. SBCs are not very powerful.
  • Misconfigured preview sizes (and lack of documentation thereof).

Even using the Preview Generator, it can take days or even weeks on a Raspberry Pi to generate the previews for a big collection which throws many people off.

Another complaint is that too many previews are generated so often the previews folder takes even more space that the pictures themselves.

The most obvious low hanging fruit to try to deal with this issue is noticing the fact that even though our systems are multi processor, only one CPU is used to generate the previews so there are plenty unutilized resources available that are not being used.

First, let’s take a quick look at the Nextcloud Previews system.

How previews work

Every time we upload an image two previews are generated

  • Small 32×32 preview. This is the icon in the Files list.
  • Max preview. This is the big picture that shows in the browser when you click on it.

The Max preview size defaults to 4096×4096, and can be configured with

This produces

There are other previews that are not generated by default but are used by other parts of Nextcloud. These will be generated on the fly unless they are pregenerated with the App.

  • The Gallery App requests (js/thumbnail.js)
    • 400×200 -> 384×256 / 256×384 (**) for regular thumbnails.
    • 200×200 -> 256×171 / 171×256 for folder thumbnails, which shows the first four square thumbnails of the directory.
  • The Files App in grid view mode requests (js/filelist.js)
    • 250×250 -> 256×256

(*) it requests 400×200 but the backend rounds it up to powers of two -> 384×256. These are examples only.

(**) depending on photo orientation

How the Preview Generator works

We can see that we need a properly configured Preview Generator or we might be generating previews that are useless since they are not going to be used.

By default the App generates all possible sizes from 32×32 to Max which is both a waste of CPU and space since as we just saw many of those won’t be used.

If we don’t want this behavior, the App allows us to configure what sizes we want to generate, but it is not documented anywhere what sizes are really needed unless you dig in the code. To configure this setting

So we end up with either people complaining that the previews take more GiB than the actual photos or people naively trying different sizes assuming that they will be used when they will most likely not be.

Recommended configuration

For the reasons stated above, I recommend the following settings for a SBC

In my opinion these should be the default settings for the Preview Generator App.

Just configuring the Preview Generator properly results in a 4x speed up and saves lots of wasted space (see results below).

Proposal

In order to try to find a way of using our CPUs more efficiently and to improve performance on low end devices, I forked Nextcloud and the Previews Generator App to create a Proof of Concept implementation that I could run some tests on.

We don’t need very good quality using computationally expensive filters for the small thumbnails, maybe only for the Max preview, so the proposal plays with scaling without interpolation and parallel generation to try to achieve better results.

Using these modifications I was able to achieve a 5x speed up in a 4 core Raspberry Pi and a 10x speed up in my 16 core PC while retaining good image quality.

If we compare with an unconfigured Preview Generator the gain is 20x for the Raspberry Pi and 50x for the PC (see Github link).

You can check more details about these benchmark results on Github.

Author: nachoparker

Humbly sharing things that I find useful [ github dockerhub ]

20 Comments on “Understanding and improving Nextcloud Previews

  1. “Max preview. This is the big picture that shows in the browser when you click on it.”
    ==>> Do you mean that when playing a slide show, the pictures shown are NOT the actual pictures but generated ones ????
    If so, what a wast of power and space.
    And I understand why, with fiber connection at home (on a Rock 64), it take ages to display pictures when I’m away.

  2. This is great, thanks so much for this and the work you’ve done. I will give it a try on my docker container, hopefully it will speed it up as well.

  3. Let us hope that Rullzer will include these enhancements in the official app. 🙂

    Just one thought. When running php-fpm, then the server does spawn several threads to generate thumbnails. Still extremely slow.

  4. hello,

    where and how can i change the directorie’s to preview, not every directory should be shown as a preview or only 2 to 3 specific directories would be in this site !? thanks for your help

  5. I am not quite sure I understand the commands above right 🙁

    First of all, should I use the commands in the two boxes in the “How the Preview Generator works” section or are those just explanatory?
    Second of all, when I ssh into my FreeNAS nextcloud jail and try to run occ config:app:set previewgenerator squareSizes –value=”32 256″ I get an error occ: Command not found.

    What am I doing wrong? Thanks in advance for your help!

  6. Thank you for your wonderful post. However there seems to be another bug which makes Nextcloud super slow: it is trying to make previews of mp3-s which takes minutes for just one mp3. I have a fresh Nextcloudpi setup (on a Raspi 3), uploaded a directory with a few mp3-s using webdav, and then visit that directory with the Nextcloud ios app without clicking on any file. These lines appear in /var/log/apache2/nc-access.log:

    10.10.11.177 – – [21/Jul/2019:10:34:34 +0100] “GET /index.php/core/preview.png?file=body%20mind%20soul/bl%20purpose.mp3&x=576&y=768&a=1&mode=cover HTTP/2.0” 404 0 “-” “Mozilla/5.0 (iOS) Nextcloud-iOS/2.23.7” ~126/126258422~
    10.10.11.177 – – [21/Jul/2019:10:34:34 +0100] “GET /index.php/core/preview.png?file=body%20mind%20soul/bl%20ml%201of2.mp3&x=576&y=768&a=1&mode=cover HTTP/2.0” 404 0 “-” “Mozilla/5.0 (iOS) Nextcloud-iOS/2.23.7” ~126/126590957~
    10.10.11.177 – – [21/Jul/2019:10:34:34 +0100] “GET /index.php/core/preview.png?file=body%20mind%20soul/bl%20med.mp3&x=576&y=768&a=1&mode=cover HTTP/2.0” 404 0 “-” “Mozilla/5.0 (iOS) Nextcloud-iOS/2.23.7” ~126/126603509~
    10.10.11.177 – – [21/Jul/2019:10:34:34 +0100] “GET /index.php/core/preview.png?file=body%20mind%20soul/bl%20law%20of%20life.mp3&x=576&y=768&a=1&mode=cover HTTP/2.0” 404 0 “-” “Mozilla/5.0 (iOS) Nextcloud-iOS/2.23.7” ~126/126610499~

    ~126/126610499~ means that it ran 126 seconds (and 126610499 us to be precise)

  7. Hello!
    Is it possible to generate previews for videos like .mp4 or even .gif?
    Im using the cronjob from the nextcloudpi webpanel, pictures are no problem its really smooth with my rpi 4.

    In my gallery i got 4 folders filled with pictures. But there should be 3 more folders filled with videos and gifs.
    One of the working gallery folders got 1 gif data and is even pre generated. I just cant understand why the other folders aren’t pre generated.

    Would be nice if someone could help me, i saw that this site is really active in the comments!

    Ty and have a nice day!
    Julian

  8. Very Cool! Thanks, after setting app as here written and “reset” the preview cache, there are following files were generated:
    /preview/1020200:
    total 444
    drwxrwxr-x 2 www-data www-data 4096 Jul 24 21:26 .
    drwxrwxr-x 13403 www-data www-data 274432 Jul 25 09:03 ..
    -rw-rw-r– 1 www-data www-data 128154 Jul 24 21:26 1080-1920-max.jpg
    -rw-rw-r– 1 www-data www-data 6133 Jul 24 21:26 144-256.jpg
    -rw-rw-r– 1 www-data www-data 9542 Jul 24 21:26 256-256-crop.jpg
    -rw-rw-r– 1 www-data www-data 14211 Jul 24 21:26 256-455.jpg
    -rw-rw-r– 1 www-data www-data 1618 Jul 24 21:26 64-64-crop.jpg

    OLD Preview Folder is bigger and has less files in it:
    preview/1020200:
    total 1228
    drwxr-xr-x 2 www-data www-data 4096 Jan 14 2019 .
    drwxrwxr-x 39159 www-data www-data 933888 Jul 24 14:00 ..
    -rw-r–r– 1 www-data www-data 309785 Jan 14 2019 1080-1920-max.jpg
    -rw-r–r– 1 www-data www-data 1279 Jan 14 2019 32-32-crop.jpg

  9. Thanks so much for tracking down why Nextcloud can sometimes be so very painfully slow an SBC’s. SBC’s being constrained hardware will indeed reveal where the obvious wastages and bottlenecks and inefficiencies are. Thanks for all your awesome contributions, NachoParker!

  10. Thanks, very insightful!

    I did notice that on my Nextcloud (not NextCloudPi) instance, 1024×1024 thumbs are requested for the 32×32 images in the files list. So with your recommended settings, it’s still creating 1024×1024 thumbs on the fly. Is this configurable somewhere? Or is this different for NextCloudPi vs regular Nextcloud? In any case with 1024 added to the squareSizes setting, no thumbs are created on the fly anymore so that’s great!

Leave a Reply

Your email address will not be published. Required fields are marked *