Package com.isomorphic.datasource

# Class ParquetDataSource

java.lang.Object
com.isomorphic.base.Base
com.isomorphic.datasource.DataSource
com.isomorphic.datasource.BasicDataSource
com.isomorphic.datasource.ParquetDataSource

All Implemented Interfaces:
  `com.isomorphic.base.IAutoConfigurable`, `com.isomorphic.datasource.Committable`, `com.isomorphic.datasource.FreeResourcesHandler`, `com.isomorphic.datasource.IType`, `IToJSON`, `Serializable`

public class ParquetDataSource extends BasicDataSource

A Parquet-backed `DataSource` implementation (serverType="parquet") that can:

1. Serve Parquet data via SmartClient DSRequests (currently `executeFetch(DSRequest)` delegates to `executeParquetFetch(DSRequest)`).
2. Generate SQL import artifacts from a Parquet file, including a SQL `.ds.xml` and (optionally) a SmartClient test-data file (`.data.xml` or JSON).

## Parquet source location (`dataURL`)

This DataSource reads Parquet data from the `dataURL` property in the DataSource configuration. Supported forms:

- webRoot-relative path (no scheme): `examples/shared/data/myfile.parquet`
- file:// (treated as webRoot-relative per SmartClient example conventions): `file:/examples/shared/data/myfile.parquet` or `file://examples/shared/data/myfile.parquet`
- https:// URL: `https://example.com/myfile.parquet`

## Schema auto-derivation (`autoDeriveSchema:true`)

If `autoDeriveSchema:true` is provided and no explicit `fields` are defined (or if `appendMissingColumns` behavior is enabled), this class derives field definitions from the Parquet file schema referenced by `dataURL`. Field titles are humanized and primitive Parquet types are mapped to SmartClient field types. Group/complex types default to `"text"`.

## SQL artifact generation APIs

These APIs are intended for quickly generating SmartClient SQL DataSource artifacts that can be used to import or test Parquet data with SQLDataSource tooling.

- `getSQLDSXML()` / `getSQLDSXML(String, String, String)`: Generates a SQL `.ds.xml` (serverType="sql") as a String (no filesystem writes). This is useful when the caller wants to create a disposable SQL DS via `DataSource.fromXML(xml)`.
- `createSQLDS()` / `createSQLDS(String, String, String, String, String, Integer)`: Writes a SQL `<dsId>.ds.xml` under `webRoot` and also writes a data file for test/import.
  Filesystem access required: these methods resolve `webRoot` from `Config.getGlobal()` and write output files under that directory.
- copyData(String, CopyDataFormat, Integer, Integer, boolean) / copyData(String, String, String, CopyDataFormat, Integer, Integer, boolean): Copies Parquet rows to SmartClient test-data XML (`.data.xml`) or JSON. Supports streaming output and optional batching (split output into multiple files after N rows).

See Also:
  - `executeFetch(DSRequest)`
  - `createSQLDS(String, String, String, String, String, Integer)`
  - `getSQLDSXML(String, String, String)`
  - Serialized Form

- ## Field Summary

  ### Fields inherited from class com.isomorphic.datasource.DataSource

  `OP_ADD, OP_CLIENT_EXPORT, OP_CUSTOM, OP_DOWNLOAD_FILE, OP_FETCH, OP_GET_FILE, OP_GET_FILE_VERSION, OP_HAS_FILE, OP_HAS_FILE_VERSION, OP_LIST_FILE_VERSIONS, OP_LIST_FILES, OP_LOAD_SCHEMA, OP_REMOVE, OP_REMOVE_FILE, OP_REMOVE_FILE_VERSION, OP_RENAME_FILE, OP_SAVE_FILE, OP_UNIQUE_NAME, OP_UPDATE, OP_VALIDATE, OP_VIEW_FILE`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `com.isomorphic.datasource.ParquetDataSource.CreateSQLDSResult`
  `createSQLDS()`
  Convenience wrapper that reads `dataURL` from this DataSource's configuration (typically `this.dsConfig.get("dataURL")`) and then delegates to `createSQLDS(String, String, String, String, String, Integer)`.
  `com.isomorphic.datasource.ParquetDataSource.CreateSQLDSResult`
  `createSQLDS(String dataURL, String explicitDsId, String explicitTableName, String dsOutputRelDir, String dataOutputRelDir, Integer maxRows)`
  Generates SmartClient SQL import artifacts from a Parquet file: Writes `<dsId>.ds.xml` (serverType="sql") Writes `<recordTag>.data.xml` containing rows read from Parquet
  `String`
  `getSQLDSXML()`
  Generates SQL DataSource XML (serverType="sql") for the Parquet file referenced by this DataSource instance (via `dataURL` in `dsConfig`).
  `String`
  `getSQLDSXML(String dataURL, String explicitDsId, String explicitTableName)`
  Generates SQL DataSource XML (serverType="sql") for the given Parquet location.

  ### Methods inherited from class com.isomorphic.datasource.BasicDataSource

  `getAuditDataSource, getAuditRecord, getAuditRecord, getFTSEligibleFields, getFTSProviderForField, getFullTextSearchConfig, hasCustomDefaultFetchOperation, hasCustomLogic, isFieldFTSEligible, isFullTextSearchEnabled, shouldAutoJoinTransaction, shouldAutoStartTransaction, transformMultipleFields, transformMultipleFields, writeMultiAudits`

  ### Methods inherited from class com.isomorphic.datasource.DataSource

  `add, add, addDynamicDSGenerator, addDynamicDSGenerator, addDynamicDSGenerator, clearDynamicDSGenerators, convertRelativeDates, convertRelativeDates, convertRelativeDates, convertRelativeDates, convertRelativeDates, convertRelativeDates, execute, executeAdd, executeCustom, executeFileSource, executeRemove, executeUpdate, fetch, fetch, fetchById, fetchById, fetchById, fetchSingle, fetchSingle, filter, filter, fromXML, fromXML, fromXML, fromXML, fromXML, fromXML, fromXML, fromXML, get1ManyRelationFields, getAuditChangedFieldsFieldName, getAuditRevisionFieldName, getAuditTimestampFieldName, getAuditTypeFieldName, getAuditUserFieldName, getDefaultDynamicDSGenerator, getDirectFields, getDynamicDSGenerators, getEnumConstantProperty, getEnumOrdinalProperty, getEnumTranslateStrategy, getFieldNames, getFile, getFileAsInputStream, getFileAsString, getFileContentsField, getFileFormatField, getFileNameField, getFileTypeField, getID, getJoinDSName, getJoinDSName, getListProperties, getListProperties, getListProperty, getManyManyRelationFields, getMapProperty, getName, getNonIncludedFields, getObjectProperty, getOperationProperty, getPrimaryKey, getProperties, getProperties, getProperties, getProperty, getPropertyJavaClass, getRecordXPath, getRelatedDisplayRecord, getRelatedDisplayRecord, getRelatedDSName, getRelatedDSName, getRelatedTableAlias, getRelationFields, getStringInBrowser, getTableName, getTransactionObject, getTransactionObject, has1ManyRelationFields, hasFile, hasManyManyRelationFields, hasRecord, hasRecord, hasRelationFields, initialized, is1ManyRelationField, isExpensive, isExpensive, isManyManyRelationField, isModificationOperation, isRelationField, isServerOnly, listFiles, listFiles, listFiles, listFiles, listFiles, listFiles, remove, removeDynamicDSGenerator, removeDynamicDSGenerator, removeDynamicDSGenerator, removeFile, renameFile, saveFile, setEnumConstantProperty, setEnumOrdinalProperty, setEnumTranslateStrategy, setOmitNullMapValuesInResponse, setProperties, transformImportValue, transformImportValue, transformResponse, update, update, validate, validate, validate, validateRecord`

- ## Method Details

  - ### createSQLDS

    public com.isomorphic.datasource.ParquetDataSource.CreateSQLDSResult createSQLDS() throws Exception
    Convenience wrapper that reads `dataURL` from this DataSource's configuration (typically `this.dsConfig.get("dataURL")`) and then delegates to `createSQLDS(String, String, String, String, String, Integer)`.

    Filesystem access required: this method writes generated files under `webRoot`.

    Returns:
      Map containing derived ids and output file paths (for example: dsId, tableName, recordTag, dsXmlPath, dataXmlPath, rowsWritten).
    Throws:
      `Exception` - on any read/write/parse error

  - ### createSQLDS

    public com.isomorphic.datasource.ParquetDataSource.CreateSQLDSResult createSQLDS(String dataURL, String explicitDsId, String explicitTableName, String dsOutputRelDir, String dataOutputRelDir, Integer maxRows) throws Exception
    Generates SmartClient SQL import artifacts from a Parquet file:
    - Writes `<dsId>.ds.xml` (serverType="sql")
    - Writes `<recordTag>.data.xml` containing rows read from Parquet

    Filesystem access required: resolves `webRoot` from global Config and writes output files under that directory. If `webRoot` is not set, this method throws an `IllegalStateException`.

    Parameters:
      `dataURL` - parquet location (required)
      `explicitDsId` - optional DS id (if null, derived from URL)
      `explicitTableName` - optional tableName (if null, defaults to dsId)
      `dsOutputRelDir` - optional output dir for .ds.xml (relative to webRoot)
      `dataOutputRelDir` - optional output dir for .data.xml (relative to webRoot)
      `maxRows` - optional row limit (null or <0 means all rows)
    Returns:
      Map containing derived ids and output file paths (dsXmlPath, dataXmlPath) and rowsWritten.
    Throws:
      `Exception` - on any read/write/parse error

  - ### getSQLDSXML

    public String getSQLDSXML() throws Exception
    Generates SQL DataSource XML (serverType="sql") for the Parquet file referenced by this DataSource instance (via `dataURL` in `dsConfig`).

    This method performs no filesystem writes. It reads the Parquet schema and returns the generated DataSource XML as a String so the caller can do `DataSource.fromXML(xml)` to create a disposable SQL DataSource.

    Conventions:

    - `dsId` defaults to `this.dsName` when present, otherwise derived from the Parquet filename
    - `tableName` defaults to `dsId`
    - `testFileName` is omitted by default (no test harness). If you want it, call the overload `getSQLDSXML(String, String, String)` and pass a value.
    Returns:
      DataSource XML as a String (serverType="sql")
    Throws:
      `Exception` - if `dataURL` is missing/invalid or Parquet schema cannot be read

  - ### getSQLDSXML

    public String getSQLDSXML(String dataURL, String explicitDsId, String explicitTableName) throws Exception
    Generates SQL DataSource XML (serverType="sql") for the given Parquet location.

    This method performs no filesystem writes. It reads the Parquet schema and returns the generated DataSource XML as a String.

    Parameters:
      `dataURL` - parquet location (required)
      `explicitDsId` - optional DS id override (if null, derived from URL)
      `explicitTableName` - optional tableName override (if null, defaults to dsId)
    Returns:
      DataSource XML as a String (serverType="sql")
    Throws:
      `Exception` - on any read/parse error