Archiving

Applies to version: 2022.1.x and above; author: Łukasz Maciaszkiewicz

Introduction

Increasing data volumes combined with shrinking disk space and decreased performance pose a challenge to every IT system. WEBCON BPS faces it with an extensive archiving mechanism that automatically removes data from process databases. The resulting better arrangement of disk space translates into enhanced performance and reliability of the whole system.

This article describes the aforementioned archiving mechanism in WEBCON BPS and includes information on its configuration.

NOTE: archiving should not be confused with backup – archived forms or attachments cannot be restored to the operational database. Archiving is a one-way, irreversible operation.

Archiving mechanism in WEBCON BPS

In WEBCON BPS a user specifies processes to undergo archiving. This means that archiving does not affect the application in which an archived process is located nor other processes grouped in it.  Archiving related processes, i.e. parent and child, looks different in that respect. Parent process archiving involves child process as well. In the opposite situation, once a child process is archived, the parent behaves as if it never had a child.

It is worth to mention here the types of data that do not undergo archiving. Those essentially include all dynamic content, such as values in “Data table”, “Data row”, “Chart”, or “HTML” form fields. Additionally, archiving does not include custom controls, as well as form and business rules.

The visibility of an archived instance is another important matter. In such case all the fields visible just before the instance is moved from the content database will be visible after archiving (if some form fields were hidden on the field matrix, they are not visible in the archive view). Moreover, although the general layout of the instance corresponds to the field matrix configuration, the aforementioned dynamic content is not included. Also, note that changing form field visibility (on the matrix) does not affect its visibility in the archive view. 

The comparison of instance view before and after archiving. The items captured within the frame: (1) “Person or group”, “Single line of text”, and “Floating-point number” form fields; (2) “Data table” form field including dynamic content.

Configuration

• Database

Depending on the selected level of archiving, it might be necessary to create an archive database the data will be moved to. It is possible during WEBCON BPS installation, but also later using the WEBCON BPS installer. After its running, select “Tools for application management”.

The “Tools for application management” option available in the installer window allows users to move to database creation.

Then, go to the “Database creation” step available on the left, select “Archive Databases” (related to specific content database), and click “Create archive database”.

The “Database creation” node and the window it opens

After clicking the button, it is possible to define the database parameters, such as name and user (obtains owner privileges for the created database). Once you fill out all the required fields and click the “Create” button, the installer shows a summary of parameters defined for the database. Confirming them starts the database creation. Once the database is created, a corresponding message appears and shows its basic data.

The field allows users to define the database parameters

If there is more than one archive database created, then it is possible to define which database should be the default one. To do that click “System settings” in WEBCON BPS Designer Studio, select the “Database” node, and choose the database in the “Archive database” group. The database indicated here is used by all the processes for the “Archive workflow instances” action unless a different database is defined for a given process. You can attribute a dedicated archive database to the selected processes in the process configuration in the “Settings” tab.

Default archive database configuration

• Database structure

The structure of archive database resembles a simplified version of process database. It includes the following tables:

• GlobalParameters – stores general information on the archive database;
• WFArchivedElementDatas – includes XML files with information on specific instance and its history;
• WFAttachmentFiles – saves archived attachment files;
• WFDataAttachmets – provides information on archived attachments;
• WFElementDetails – stores data from archived instance item list;
• WFElements – saves general information on archived instance, such as document ID, instance number, form field values, except item list, etc. It is used only to display report data. The value on the form is taken from the XML file of the archived instance.
• WFSecurities – stores information on user privileges to archived elements.

• Service role

The creation of archive database itself is not enough to ensure correct operation of the archiving action. To be able to process the archive queue correctly the service needs to be assigned an additional role in its configuration. To do that, go to “System settings” and check the “Workflow instance archiving” checkbox in the service configuration. After saving the settings it is necessary to reload the service configuration. If the role is not enabled or the service configuration is not reloaded, the archiving action will not work.

Enabling the “Workflow instance archiving” service role

• Archiving threads

The archiving actions can be run in multiple threads, even though the default configuration is one thread. This configuration can be modified in the service configuration. To do that go to “System settings”, expand the “Services configuration” node, select “Services”, and choose the configuration of a service responsible for archiving. The “Configuration” tab includes the “Number of simultaneous archiving threads” item. After changing the default number of threads click “Save”. However, please remember that the number of threads can negatively affect system performance.

Changing the number of archiving threads

• Actions

The archiving mechanism embedded in WEBCON BPS allows users to use two archiving actions:

1. Archive workflow instances – the action is used to archive selected workflow instances. It is possible to run the action in the following contexts: “On entry”, “On exit”, “On timeout”, “Menu button”, “On path”, “Upon instance saving”, and “On start (cyclical)”.
2. Archived instances retention – the action is used to execute operations on the already archived workflow instances. Except for the “Upon instance saving”, the action is triggered in the same contexts as the ones described for “Archive workflow instances”.

The basic difference between these actions lies in the database used for the operations and the number of retention modes available for configuration. For the first action all the three modes stipulated below are available, whereas the action “Archived instances retention” can be triggered in two modes: “Move to external network location” and “Remove from content database”.

• Retention modes

The archiving process can be executed in three modes. Irrespective of the selected mode the data are irreversibly deleted from the process database. It only retains information on the removed instances numbers in order to ensure uniqueness – the instance number can never repeat. The retention modes are described below:

1. Move to archive database – the workflow instances specified in the archiving action are moved to a separate, dedicated database. Following this operation, the archived documents are only available in the read-only mode. They can be viewed using a properly configured report. The mode allows users to view all historical data and associated attachments.
2. Move to external network location – the specified workflow instances are moved to a defined network location in ZIP files. The ZIP package consists of a PDF file with metadata on the form state preceding the archiving and an XML file including instances history with attachments. Subworkflow instances can be stored in separate packages (one subworkflow – one package) or in one package with the main workflow instances. To use this option it is necessary to define the network location in the process configuration where the data package with files is exported.
3. Remove from content database – the configuration is similar to the configuration of the first mode, but the workflow instances are irreversibly removed the operational database. No copy of an archived instance is saved in this case.

To set up a given retention mode add an action to an automatization, click the vertical ellipsis icon in the action block, and select “Configuration” from the drop-down menu. In the new window, in the “Basic settings” section expand the drop-down list in the “Retention mode” field, and select one of the afore-mentioned modes. The option “Packaging mode” is only available for the “Move to external network location” mode. In the “Instances selection criteria” you can determine how archived elements are chosen. Using the drop-down list available in “Instances selection mode”, you can choose only the current instance to undergo archiving (“Archive only current workflow instance”) or use an SQL query to indicate the instances to be archived (“Select workflow instances using SQL”). The latter option requires inserting an SQL query in the “SQL Query” field.

The options available for action configuration are discussed below.

“Archive workflow instances” configuration

Retention mode – the field allows users to select one of the retention modes described above.

Packaging mode – when the “Move to external network location” mode is selected, it is possible to choose one out of the two available modes for creation of ZIP packages:

• Instances of the main workflow and its subworkflows in one package – both the main workflow and subworkflows instances are exported to one common ZIP package.
• Instances of the main workflow and each subworkflow packaged separately – the main workflow instances and instances belonging to all the subworkflows are exported to separate ZIP packages (one subworkflow to one package).

Priority – specifies the sequence of instance archiving. The available range is: 1–10. Number “1” instances are processed as the first ones, whereas the ones with number „10” priority as the last ones. The “Nighttime” option is also available. After selecting this option the priority is automatically set to “11” value. The instances bearing such priority are processed only during night hours that can be configured in the services configuration in the “Schedules” node (“System settings” → “Services configuration” → “Schedules). The current status of the instances in the archive queue can be checked in “Archive workflow instances queue” (“Reports” → “Basic reports” → “Archive workflow instances queue”). It is recommended to run archiving in night hours to minimize the risk of decreasing system performance.

Instances selection criteria – the section allows for defining an SQL query returning the data to undergo archiving. The first, result column should return the ID of the instances to be archived. The query is executed within the default database. When an action is used in a cycle, it is not possible to use the constants operating within the context of an instance. Additionally, in the case of large amount of data it is recommended to set up the SQL query to return data in several smaller packages by using the “TOP” clause. When writing an SQL query, remember that the ID of returned instances cannot be the same as the ID of instances in the archive queue, otherwise the action execution will be unsuccessful. To avoid that situation, the T-SQL query should refer to the “ArchiveQueueItems” table that includes current information on the queue. The “ARQ_WFDID” column stores the ID of the processed instance, and thus is a foreign key referring to the “WFD_ID” column in the “WFEelements” table.

Archiving actions triggered on the form can add an instance to the archive queue in the context of the current element. This allows users to create cyclical actions moving process instances from step “A” to step “B” and trigger on path a single action that adds an instance to the archive queue.

NOTE: archiving is a one-way, irreversible operation – the archived instances cannot be restored to the process database. To avoid potential damages caused by an SQL query with a wrong filter, test how the query selects instances to archive.

Archive queue

NOTE: archiving can be a long process that burdens server. When scheduling archiving it is advised to plan it outside regular working hours, e.g. during nighttime, weekends, etc. What is more, when archiving large volumes of data it is best to use separate packages.

Instances archiving is executed asynchronously. This means that archiving actions add instances to the queue that is later processed by the service. It is possible to view the current archive queue in BPS Designer Studio in the “Reports” section (“Basic reports” node → “Archive workflow instances queue”) or by executing a T-SQL query relating to the ArchiveQueueItems table. The report includes basic information on the processed instance and the number of undertaken processing attempts. The maximum number of processing attempts is five. First attempt is executed straight away in accordance with the service interval, the second one after next service interval, the third one after 5 minutes, the fourth one after an hour, and the last one after 24 hours. If an instance cannot be archived after that time, the queue stops processing it, and its status is set to error. Processing an instance in one attempt is also time-restricted and cannot exceed 300 s. If an instance is not archived within that time, next attempt to process it is taken. This value is fixed and it is not possible to change it in WEBCON BPS Designer Studio. Advanced MS SQL users can update that value in MS SQL Management Studio. The value is stored in configuration database in the “Services” table in the “S_ArchivingDeleteTimeout” column. The report also includes “Reset”, “Delete”, and “Edit” buttons. The first one allows users to reset the number of undertaken attempts to “0”, which results in restarting instance processing. The “Delete” button allows users to completely remove an instance from the queue and withdraw it from further archiving. The last button, i.e. “Edit”, allows for changing the processing priority which is by default set by the archiving action. The whole archiving mechanism is based on three operations:

• • listing data for archiving by the action;
  • adding instances to the queue;
  • removing them from the main content database.

All three operations, i.e. SELECT, INSERT and DELETE are executed in one transaction.

The archive instance queue in the “Reports” section including two instances for archiving

Reports

WEBCON BPS also provides functionality allowing for reporting archived instances. To do that, in WEBCON BPS Designer Studio select respective application, go to the “Reports” tab in the “Presentation” section and click the “Add” button. Enter name in the newly created report and go to the “Configuration” tab. Here, in the “Process” section select “Archived” from the drop-down list and choose the process for which archived data are to be displayed. Like in the regular report, the remaining tabs allow users to define displayed columns, group, sort, etc. In contrast to the regular report, the archive instances report offers fewer options. The first difference is the lack of assignment filter and predefined filter, as it is not possible to create any user tasks on the archived instances. Missing “Mass actions” tab is another difference resulting from the fact that these instances are available only in the read-only mode and no operations can be executed on them. Additionally, the “Views” tab does not allow here to add buttons to start new instances in the upper report panel. All business and process administrators have access to the archived elements. What is more, it is possible to add more persons with access at administrator level to all the archived instances by modifying respective process settings (process name → “User privileges” → “General” → “Manage archive”). The archived instances are not indexed by SOLR, so they cannot be found by means of the main search engine. These can be found only through archived instances report.

Adding a report on the archived instances

Comparison of an archived instances report and a regular report

• Storing vs displaying archived instance data

There is a difference between the source for presentation of archived forms on the report and for presentation directly on the form. When a user opens a report presenting archived instances, the system loads data from archive database by means of an SQL query (the query can be viewed in the diagnostic mode just like in the regular report).

The system behaves differently when opening a specific archived instance form, i.e. it unparses an XML file located in the database. The XML files are stored in the “WFArchivedElementDatas” table and the column used for form identification is “WAE_WFDID”. The XML file with the form data is stored in the “WAR_ArchivedElement” column, while archived history is available from the “WAE_ArchivedHistory” column. The files can be downloaded after entering the archived form. There are two buttons available from the form menu:

• “Last version data”;
• “Historical data”.

• Privileges to archived instances reports

Business administrators have access to all the archived workflow instances. Such privileges can be also granted to specific users by entering their data in the “Manage archive” field in the “User privileges” tab of a given process (selection tree → application name → process name → “User privileges” → “Manage archive”). Note here that the field is only editable when an archive database is specified, and the persons defined in it can view documents already present in the archive database. What is important, moving an instance from operational database to archive database involves withdrawal of certain users privileges to specific instances. This means that a person holding privileges to a given instance in the operational database looses them following its archiving. Granting privileges to a specific archived instance is possible at the form level. To do that, click the gear icon (“Admin actions”) in the right upper corner and select “Privileges” from the drop-down menu. Apart from being able to view users with privileges to an instance, you can also add them in the “Add access privilege” field. It is important to note here that an author of documents does not have access to instances in the archive database and the other way around – a person privileged to manage the archive does not have access to instances in the operational database.

Summary

The archiving mechanism implemented in WEBCON BPS guarantees smooth management of instances and disk space. Following the information included in this article it is possible to configure it in a way that ensures high performance of the whole system and enhanced management of increasing volumes of data gathered over time.