Big Objects

Taking Big Objects Off-Platform

Salesforce Big Objects is an Apache HBase storage technology owned by Salesforce with subscription access to customers. Customers utilize Big Objects for greater economies of scale in comparison to native storage within Salesforce. The issues with Salesforce Big Objects include a massively reduced featureset and a negative user experience when consuming the data.

By migrating data from Salesforce Big Objects to GRAX, you can retain full ownership while taking advantage of the GRAX application layer for better visualization, organization, and data handling alongside the rest of your GRAX dataset.

Below is how your archived data will appear in the GRAX Lightning Data Viewer:

89dad46-Screen_Shot_2022-01-03_at_9.55.41_AM.png89dad46-Screen_Shot_2022-01-03_at_9.55.41_AM.png

CRITICAL: Complete Full Backup + Nightly Incremental BEFORE Archiving ANY Data

It is critical you complete a full-org Backup and schedule full-org nightly incremental backups before proceeding. Salesforce Cascading Deletes, Delete Triggers, and/or Custom Processes you can NEVER assume hierarchy archive secures all data. The only way to protect all data is via a full-org Backup and full-org nightly incremental backups.

Archiving Big Objects

GRAX supports Big Objects (BO) data out of the box, mitigating risk to existing business processes. With GRAX, you can quickly migrate all data from Big Objects and still retain 100% access within Salesforce. We recommend performing the below steps first in a Sandbox for validation and then deploying objects into production following standard operating procedures.

There are two main techniques:

Option 1: Utilize Existing Object to Archive Data

This option retains the most power with visualization, reporting, analytics, restoration, and visibility within Salesforce. The most important thing to consider is mitigating risk to/from existing business processes, integrations, workflows, and custom logic. Please ensure to triple check email notifications, triggers, process builder, workflows, external integrations, and all automated code/logic that interacts with data.

Private Sharing Model & Sharing Knowledge

Below assumes you understand sharing Here, private objects, sharing rules, how record access can be granted, and how the business currently utilizes private sharing models. Below is a summarized "sample" and should be adjusted to your custom business processes.

Defining Segmenting / Partitioning Data Strategy

It is sometimes critical to define logic for segmenting transitional Archive data from existing business processes, reports, dashboards, and users. This is done as you move data from BO to GRAX to hide the data to mitigate risks to existing business processes. It is up to the customer to define the strategy.

Sample Partitioning Strategy

Below is a sample how you could choose to partition data from the existing business data. The below segmentation strategy is based on out-of-the-box Salesforce data partitioning using record types and sharing rules.

Cautious of Record Access and Security Permissions

There are many ways to grant/prevent/revoke record access; please watch. It is 100% up to the customer to define the record access strategy and how you prevent business processes from firing. This is typically done with Ownership and Record Type but is 100% dependent on how existing deployment is configured.

Be very careful of the View All Data permission set for integration users, reporting users, and existing business processes.

  1. GRAX User - It is critical that you configure the GRAX User to use a dedicated Salesforce user so any insertion/deletion/update logic can be available for the Salesforce Admin.

  2. Create GRAX Archive Record Type - If you need to hide the data from the existing business users/reports/dashboards you could utilize a new Salesforce Record Type and only give access to the record type to the Salesforce Admin and the GRAX user.

  3. Storing/Maintaining Existing Record Type Field - It will be critical that you retain the existing record type from BO in a new field on the object for later inserting restoration logic. This can be accomplished simply by adding a new text field and only giving access to that field to the GRAX and Salesforce admin users. When you insert data from BO into the existing object (Testing & Validating, Step 7), you will map the existing record type into the new field and set the new record type to "Archive Record Type".

  4. Storing/Maintaining Previous Ownership - It will be critical that you retain the existing ownership from BO in a new field on the object for later inserting restoration logic. This can be accomplished simply by adding a new text field and ONLY giving access to that field to the GRAX and the Salesforce admin user. When you insert data from BO into the existing object (Testing & Validating Step 7) you will map the existing ownership into the new field and set the new record ownership to GRAX (for driving privacy and sharing rules).

  5. Testing & Validating - Once you are ready to begin testing, skip to "Testing & Validating" below.

Option 2: Creating a New Object to Archive Data

This option is the least risk to existing business processes, reports, dashboards, integrations, workflows, and business logic. With this option you will need to add logic to restore/duplicate data from the new object to the existing. You will also lose some UI features. It is recommended that this approach only be used when you completely rule out utilizing the existing object.

In an effort to mitigate risks with existing business processes, object changes, data type changes, data validation rules, technical implementation (triggers, validation, process builder, flows, email notifications, etc.), we recommend creating a new Salesforce Object (ex. CaseHistory__c). This object will only be used as a transitive store from Salesforce into GRAX or when you restore data back into Salesforce.

The quickest way is to copy the existing object definition using SFDX, copy/rename, then save the new Object.

  • We recommend disabling “Requiring” Field Data on all Fields. (This mitigates old object data from Big Objects as business logic changes.)

  • If field types were/are different in Big Objects, change field types to String (accept everything). Data Selection from BO should apply type conversion in query and insertion into a table. The path with the least risk is changing all fields to TEXT/String on the new Object in Salesforce besides lookup values.

  • Field Validation rules should be removed to mitigate old data issues.

  • Field Lookups- Ensure all lookup fields and field level security are created correctly (they drive GRAX LWC).

  • Retaining Original Record Instance Id - If you need to retain the original record id from the original Salesforce record we highly recommend creating a text field (lookup will not work if data was removed) and writing the original value to this field. This is standard for compliance reasons.

  • Retain System Dates/Times - Decide if you will insert/keep system date/time like LastModifiedByDate, etc. You absolutely can retain this information; this is why we also recommend the new Object process. It is imperative that the user can "Create Audit Fields".

Testing and Validation

  1. Manually Create Test Data - Using Salesforce, manually create test records in the existing object (1a) or New Object (1b).

  2. Add GRAX LWC to Page Layout - Add GRAX LWC to the parent of one of the lookup fields. Add list with “All Data,” which will include backup/archived data.

  3. Create and/or Run GRAX Incremental Backup - Create an Incremental GRAX Backup job with JUST the Existing Object (1a) or New Object (1b), Execute.

  4. Validate - Return to the detail page of the parent of the lookup field on the test data you created.

  5. Create & Run GRAX Archive - Create an Incremental GRAX Archive job with JUST the Existing Object (1a) or New Object (1b), Execute. Quickest way is to build a Salesforce report and utilize that report and filter to archive. TRIPLE CHECK your report filter logic. Simpler Archive the Better!

  6. Validate - Return to the detail page of the parent of the lookup field on the test data you created. Validate the record was Archived/Removed from the New Object you Created from the First Step.

  7. Fill Table FROM Big Objects - Fill the table from Big Objects in chunks you define as a realistic mitigating risk for the business. Manually spot-check data in the Existing Object (1a) or New Object (1b), ensuring lookup values relate properly to existing data, etc.

    • RECOMMENDATION - Triple check the record types (1a or 1b), created date retention, ownership (1a map to new field for privacy if needed), and all system fields coming from BO to Salesforce before you archive.

    • RECOMMENDATION - Query/fill data oldest to newest to expose data inconsistencies which you can modify table definition or fix/convert data. Typically the oldest data will have the largest schema drift, dirty data, and the quickest/best way to validate New Object definition.

  8. Run GRAX Archive Step #7 - Start small and repeat.

  9. Repeat and/or Automate via Scheduled Job Step #7 + #8 until complete.

FAQ

Why a New Object vs. Existing?

  • Mitigate Business Risk - Mass inserting old data into an existing object driving the current business process is way too much risk, and we DO NOT ever recommend that (Bad data, email notifications, business processes, integrations, all create risk and do not recommend).

  • Data Incompatibility - Typically, when data is put into Big Objects, data types change, bad data, and incompatibility in existing tables. The New Object allows you to modify field type definition without risking business processes.

  • Remove Required Field and Validation Rules - You will need to remove all field-level requirements and validation rules to mitigate old data or fix data in the query from Big Objects.

  • 100% GRAX LWC Visibility - Once data is archived in GRAX, you will have full access and visibility within Salesforce.

  • Auditability - By retaining the original data types stored within Big Objects, you can keep an exact replica of the copy in GRAX without risking any changes you have made to the object within your Salesforce.

What do I lose using a New Object vs. Existing?

  • BENEFIT: Restore to New Object vs. Original Object - When you restore the data from GRAX it will go into the New Object vs. the Original Object, which is a sizeable benefit. Restoring to the New Object gives you a layer of abstraction to define data and business rules before adding data back into the standard object. You “could” also create a lookup relation and maintain 100% tracking and visibility of the new Object.