Skip to main content

Download Dataset

Download an existing dataset using the Knowledgator Python SDK.

The platform provides a secure HTTP 307 redirect to object storage, which the SDK resolves automatically (follow_redirects=True). The SDK offers two download modes:

  • Direct Download (kg.datasets.download): Save directly to a local file or parse into memory.
  • Streaming Download (kg.datasets.stream_download): Stream chunks iteratively with constant $O(1)$ memory consumption, ideal for large datasets.

Direct Download (kg.datasets.download)​

1. Save Directly to Disk​

Specifying file_path streams the file from storage straight into your destination file without buffering the entire payload in application memory:

from knowledgator import Knowledgator

kg = Knowledgator(api_key="YOUR_API_KEY")

saved_path = kg.datasets.download(
dataset_id="6babd48a-2256-4466-b16b-9fd45ba59688",
file_path="./data/downloaded_dataset.json"
)
print("Saved dataset to:", saved_path)

2. Parse Directly into Memory (return_data=True)​

To immediately inspect or manipulate dataset rows as Python objects without saving a local file:

from knowledgator import Knowledgator

kg = Knowledgator(api_key="YOUR_API_KEY")

samples = kg.datasets.download(
dataset_id="6babd48a-2256-4466-b16b-9fd45ba59688",
return_data=True
)

print(f"Loaded {len(samples)} samples:")
for sample in samples[:3]:
print(sample)

3. Asynchronous Direct Download​

import asyncio
from knowledgator import Knowledgator

async def main():
kg = Knowledgator(api_key="YOUR_API_KEY")
data = await kg.datasets.download_async(
dataset_id="6babd48a-2256-4466-b16b-9fd45ba59688",
file_path="./data/async_dataset.json",
return_data=True
)
print("Async download finished. Sample count:", len(data))

asyncio.run(main())

Streaming Download (kg.datasets.stream_download)​

For very large files (e.g. hundreds of megabytes or millions of records), use the streaming generator. It yields byte chunks one by one, keeping RAM usage minimal.

Synchronous Chunk Streaming​

from knowledgator import Knowledgator

kg = Knowledgator(api_key="YOUR_API_KEY")

total_bytes = 0
with open("./large_dataset.json", "wb") as f:
for chunk in kg.datasets.stream_download("6babd48a-2256-4466-b16b-9fd45ba59688", chunk_size=65536):
f.write(chunk)
total_bytes += len(chunk)

print(f"Successfully streamed {total_bytes} bytes.")

Asynchronous Chunk Streaming (stream_download_async)​

import asyncio
from knowledgator import Knowledgator

async def main():
kg = Knowledgator(api_key="YOUR_API_KEY")

total_bytes = 0
async for chunk in kg.datasets.stream_download_async("6babd48a-2256-4466-b16b-9fd45ba59688", chunk_size=65536):
total_bytes += len(chunk)

print(f"Streamed {total_bytes} bytes asynchronously.")

asyncio.run(main())

Method Parameters​

kg.datasets.download(...) / kg.datasets.download_async(...)​

ParameterTypeRequiredDescription
dataset_idstrYesTarget dataset UUID.
file_pathstr | PathNoDestination path on disk. Missing parent directories are created automatically.
return_databoolNoIf True, parses and returns the JSON content as Python structures. Defaults to False.
timeout_msintNoRequest timeout in milliseconds.

kg.datasets.stream_download(...) / kg.datasets.stream_download_async(...)​

ParameterTypeRequiredDescription
dataset_idstrYesTarget dataset UUID.
chunk_sizeintNoByte chunk size to yield (default: 65536 bytes / 64 KB).
timeout_msintNoRequest timeout in milliseconds.