SOLR indexer queue contains many entries

0

Hi Maria,
check the SOLR log first:
X:\Program Files\WEBCON\WEBCON BPS Search Server\Search Cluster\Solr\server\logs\solr.log
are there any entries about out of memory?
You can check the memory usage in SOLR admin panel:
http://localhost:8983/solr/#/ - section Dashboard, JVM Memory on the right side. Default is 2GB, you can extend it to eg. 4GB by editing X:\Program Files\WEBCON\WEBCON BPS Search Server\WebCon.BPSSearchService.exe.config if the culprit is lack of heap memory.

Thank you very much! Yes, that was indeed the issue. I've increased it to 6 GB.

The thing is, I don't see this number decreasing significantly over the past few days. There are around 800k items with `Status = ProcessingPending` and `Priority = 5`, and the count is decreasing by about 40k per day. :(

What would you suggest I do? Should I run a clear and reindex process, or reindex the entire database?

Looks like your SOLR was not indexing stuff for a while... Was your global search working at all? I would wait for the queue to offload if it goes smoothly and then check how the daily indexing looks like.

Global search isn't working for the new items because they're still stuck in the processing queue. The queue is decreasing, but quite slowly. Based on the current rate of around 40k items per day, it would take approximately 20 days to clear the remaining 800k items.

Hi Maria,

Generally speaking, every save made to a workflow instance will add (at least) two entries to SOLR indexer queue:
a) update instance data in Solr (for searching and SearchIndex report purposes)
b) register user's activity (for users' last activities)
If your Solr server was not responding for a period of time, it is possible that tasks in the queue multiplicated, eg. there are 10 tasks to update the same workflow instance (unnecessary).

One possible solution to deal with such scenario is to first remove all tasks from SOLR indexer queue and then add a new task "Reindex all databases". Instead of updating each entry in Solr X times, it will be updated once.
However, there is a risk that users' activities in queue will be lost, because they cannot be reindexed (they are stored only in Solr). If you remove them from queue before they're indexed, they will be lost forever.
Depending on your needs, this may still be a better option than waiting 20 days.
Details on Solr indexer queue options can be found in https://community.webcon.com/posts/post/report-solr-indexer-queue/274/26

Since WEBCON 2024, there are separate queues for indexing instances, attachments and activities, which means that you could remove tasks only from instances and attachments queues (probably directly on SQL database) and activities would remain unchanged (no user activities would be lost). This is not possible on older WEBCON versions, because there is one common queue for every Solr indexing task.

There are other risks related to "Reindex all databases" task to consider:
- when reindexing a process/entire database, workflow instances are batched (by default data of 500 instances is sent at once), which is generally faster, but more demanding on Solr server, so more resources may be required for the period of reindexing
- reindexing can block a current/ongoing indexation, so you may consider reindexing after business hours

If you're on version 2023 or older, I'd strongly suggest upgrading ASAP, as there were major optimizations made to Solr queues and indexing. It will help you avoid similar problems in the future.

Kind regards