Skip to main content

Download Dataset

Download an existing dataset using the Knowledgator TypeScript / JavaScript SDK.

The platform provides a secure HTTP 307 redirect to object storage, which the SDK resolves automatically. The SDK supports two primary modes:

  • Direct Download (kg.datasets.download): Save directly to a local file or parse into an in-memory object.
  • Streaming Download (kg.datasets.streamDownload): Stream chunks with an async generator (AsyncGenerator<Uint8Array>), ideal for memory efficiency and large datasets.

Direct Download (kg.datasets.download)​

1. Save Directly to Disk (filePath)​

Streams the file from storage directly into the local destination file, automatically creating parent directories if needed:

import { createClient } from '@knowledgator/typescript-sdk';

const kg = createClient();

const savedPath = await kg.datasets.download('6babd48a-2256-4466-b16b-9fd45ba59688', {
filePath: './data/dataset.json'
});
console.log('Dataset saved to:', savedPath);

2. Parse Directly into Memory (returnData: true)​

Directly parses the dataset content into a JavaScript array/object:

import { createClient } from '@knowledgator/typescript-sdk';

const kg = createClient();

const data = await kg.datasets.download('6babd48a-2256-4466-b16b-9fd45ba59688', {
returnData: true
});

console.log('Loaded samples:', Array.isArray(data) ? data.length : typeof data);

3. Retrieve Raw Bytes (Uint8Array)​

When neither filePath nor returnData is passed, the method returns the entire dataset as a Uint8Array:

import { createClient } from '@knowledgator/typescript-sdk';

const kg = createClient();

const rawBytes = await kg.datasets.download('6babd48a-2256-4466-b16b-9fd45ba59688');
console.log('Downloaded bytes:', (rawBytes as Uint8Array).length);

Streaming Download (kg.datasets.streamDownload)​

For large datasets, streamDownload returns an AsyncGenerator<Uint8Array>, allowing you to stream data chunk by chunk without loading the entire payload into RAM:

import { createClient } from '@knowledgator/typescript-sdk';
import { createWriteStream } from 'fs';

const kg = createClient();

const fileStream = createWriteStream('./large_dataset.json');

for await (const chunk of kg.datasets.streamDownload('6babd48a-2256-4466-b16b-9fd45ba59688', { chunkSize: 65536 })) {
fileStream.write(chunk);
}

fileStream.end();
console.log('Streaming download completed.');

Method Parameters​

kg.datasets.download(datasetId, options)​

ParameterTypeRequiredDescription
datasetIdstringYesTarget dataset UUID.
options.filePathstringNoLocal destination file path. Streams content directly to disk.
options.returnDatabooleanNoIf true, parses JSON content and returns a JavaScript object/array. Defaults to false.

kg.datasets.streamDownload(datasetId, options)​

ParameterTypeRequiredDescription
datasetIdstringYesTarget dataset UUID.
options.chunkSizenumberNoChunk size in bytes (defaults to 65536 / 64 KB).