External Dataset Injection

The External Dataset feature is a powerful architectural pattern in DFBE that allows you to decouple large data sources (like city lists, university catalogs, or dynamic campaigns) from your JSON schema.

🚀 1. The "Primitive State, Rich Payload" Pattern

The core engine handles data in two distinct layers to maximize performance:

1. Internal Form State (Primitive): Inside the UI and react-hook-form, the field value is stored as a simple primitive (e.g., "jkt"). This ensures that heavy UI components like dropdowns don't lag when processing massive lists.
2. Validated Payload (Rich Object): Upon submission, the engine's Zod layer automatically transforms those primitive IDs into their corresponding Full Objects (e.g., { id: "jkt", name: "Jakarta", province: "DKI" }).

⚙️ 2. Configuration Structure

Datasets are passed via the dataset prop in <FormEngineProvider>.

const externalData = {
  // 'cities' is the dataset name (identifier)
  cities: {
    // Array of field names in your schema that should use this data
    fieldId: ["home_city", "office_city"],

    // The actual array of data objects
    data: [
      { id: "jkt", name: "Jakarta", region: "Java" },
      { id: "bdg", name: "Bandung", region: "Java" },
      // ... up to 100,000+ items
    ],

    // Tells the engine which keys to use for lookup and display
    key: {
      value: "id", // The primary key (stored in form state)
      label: "name", // The display label (used by adapters)
    },
  },
};

<FormEngineProvider schema={schema} dataset={externalData}>
  <App />
</FormEngineProvider>;

🧠 3. How it Works (Under the Hood)

O(1) Lookup Optimization

When the engine initializes, it doesn't just store the array. It converts every dataset into a Map<string, unknown> where the key is your key.value converted to a string.

• Validation Speed: Even if you have 200,000 cities, looking up a city ID takes O(1) time. Validation remains instant regardless of data size.
• Memory Efficiency: The data is stored once in context and shared across all bound fields.
• Key Type: All keys are stored as string (via String() coercion). A numeric ID 42 is stored as "42".

Zod Transformation Layer

The engine automatically injects a .transform() block into your field's Zod schema:

// Simplified logic of what the engine generates
const schema = z.string().transform((id) => {
  return datasetMap.get(id) || id;
});

🛠 4. How Options Are Provided to Adapters

The engine automatically handles options resolution. FieldRenderer passes the resolved options directly via the apiOptions prop in the Adapter Interface — you do not need to access datasetLookups manually to get the list of options.

However, if you need direct O(1) lookup access (e.g., to get the full object for a currently selected ID in a custom summary), you can access the lookups:

Example: Custom Summary Component

function FieldSummaryDisplay({ fieldName, value }) {
  const { datasetLookups } = useFormEngine();

  // Access the O(1) Map for this specific field
  const lookupMap = datasetLookups?.[fieldName];

  // Look up the full object for the currently stored primitive ID
  const richObject = lookupMap?.get(String(value));

  return <span>{richObject?.name ?? value}</span>;
}

⚠️ 5. Crucial Implementation Details

Supported Field Types

Dataset injection is only supported for field types that inherently handle dynamic or large option lists. Currently, the engine explicitly supports:

• combobox
• searchable-select

Using a dataset with any other field type (e.g., standard select or radio) will result in a runtime error during render.

Multi-Field Mapping

You can map a single dataset to multiple fields. This is perfect for "Home Address" and "Office Address" sections where both need the same list of Cities.

Array Support

The engine automatically detects if a field is an array (like checkbox-group) and transforms every item in that array into its corresponding object.

Handling "Not Found"

If a user submits a value that doesn't exist in your dataset Map, the engine is designed to fail-safe by returning the original primitive value instead of undefined.

📈 6. Performance Best Practices

• Avoid React State: Do not put massive datasets (10k+ items) into a standard React useState at the page level. Pass them directly to FormEngineProvider.
• Memoization: If your dataset is fetched from an API, ensure you memoize the dataset object to prevent unnecessary re-computations of the internal Lookup Maps.
• 🚀 Limit Rendered Items (CRITICAL): Even if the engine can handle 100k items in memory, DOM rendering is expensive. Never render the entire dataset into a standard HTML <select> or list.
  • Use a virtualized list if you need to show many items.
  • Or, and more simply, limit the search results in your adapter to the first 100 items (e.g., .slice(0, 100)) while the user is typing. Rendering 1,000+ DOM nodes simultaneously will cause severe UI lag.

📦 7. Handling Transformed Data (ID to Object)

The engine follows a "Primitive State, Rich Payload" architecture to ensure performance:

1. Form State (Primitive): The internal state of the form (what you get from formMethods.getValues() or useWatch) always remains as primitive IDs (e.g., "jkt").
2. Submission Payload (Rich): The transformation from ID to Object happens exclusively during the Zod validation process.

How to get the full Objects:

✅ The Correct Way (Submission): Use formMethods.handleSubmit. The data passed to your callback will be fully transformed.

const onSubmit = (transformedData) => {
  // transformedData.city is now { id: "jkt", name: "Jakarta" }
  console.log(transformedData);
};

<button onClick={formMethods.handleSubmit(onSubmit)}>Submit</button>;

❌ The Incorrect Way: Do not use formMethods.getValues() if you need the objects. It will only return the raw IDs.

const values = formMethods.getValues();
// values.city is still "jkt" (NOT transformed)

Accessing Transformed Data via Context:

If you are building a multi-step form and need to display a summary of transformed data before final submission, you can consume validatedData from useFormEngine() after a successful handleNext() on the last step:

const { validatedData } = useFormEngine();
// validatedData contains the rich objects