Download Dataset
Download an existing dataset using the Knowledgator Python SDK.
The platform provides a secure HTTP 307 redirect to object storage, which the SDK resolves automatically (follow_redirects=True). The SDK offers two download modes:
- Direct Download (
kg.datasets.download): Save directly to a local file or parse into memory. - Streaming Download (
kg.datasets.stream_download): Stream chunks iteratively with constant $O(1)$ memory consumption, ideal for large datasets.
Direct Download (kg.datasets.download)
1. Save Directly to Disk
Specifying file_path streams the file from storage straight into your destination file without buffering the entire payload in application memory:
from knowledgator import Knowledgator
kg = Knowledgator(api_key="YOUR_API_KEY")
saved_path = kg.datasets.download(
dataset_id="6babd48a-2256-4466-b16b-9fd45ba59688",
file_path="./data/downloaded_dataset.json"
)
print("Saved dataset to:", saved_path)
2. Parse Directly into Memory (return_data=True)
To immediately inspect or manipulate dataset rows as Python objects without saving a local file:
from knowledgator import Knowledgator
kg = Knowledgator(api_key="YOUR_API_KEY")
samples = kg.datasets.download(
dataset_id="6babd48a-2256-4466-b16b-9fd45ba59688",
return_data=True
)
print(f"Loaded {len(samples)} samples:")
for sample in samples[:3]:
print(sample)
3. Asynchronous Direct Download
import asyncio
from knowledgator import Knowledgator
async def main():
kg = Knowledgator(api_key="YOUR_API_KEY")
data = await kg.datasets.download_async(
dataset_id="6babd48a-2256-4466-b16b-9fd45ba59688",
file_path="./data/async_dataset.json",
return_data=True
)
print("Async download finished. Sample count:", len(data))
asyncio.run(main())
Streaming Download (kg.datasets.stream_download)
For very large files (e.g. hundreds of megabytes or millions of records), use the streaming generator. It yields byte chunks one by one, keeping RAM usage minimal.
Synchronous Chunk Streaming
from knowledgator import Knowledgator
kg = Knowledgator(api_key="YOUR_API_KEY")
total_bytes = 0
with open("./large_dataset.json", "wb") as f:
for chunk in kg.datasets.stream_download("6babd48a-2256-4466-b16b-9fd45ba59688", chunk_size=65536):
f.write(chunk)
total_bytes += len(chunk)
print(f"Successfully streamed {total_bytes} bytes.")
Asynchronous Chunk Streaming (stream_download_async)
import asyncio
from knowledgator import Knowledgator
async def main():
kg = Knowledgator(api_key="YOUR_API_KEY")
total_bytes = 0
async for chunk in kg.datasets.stream_download_async("6babd48a-2256-4466-b16b-9fd45ba59688", chunk_size=65536):
total_bytes += len(chunk)
print(f"Streamed {total_bytes} bytes asynchronously.")
asyncio.run(main())
Method Parameters
kg.datasets.download(...) / kg.datasets.download_async(...)
| Parameter | Type | Required | Description |
|---|---|---|---|
dataset_id | str | Yes | Target dataset UUID. |
file_path | str | Path | No | Destination path on disk. Missing parent directories are created automatically. |
return_data | bool | No | If True, parses and returns the JSON content as Python structures. Defaults to False. |
timeout_ms | int | No | Request timeout in milliseconds. |
kg.datasets.stream_download(...) / kg.datasets.stream_download_async(...)
| Parameter | Type | Required | Description |
|---|---|---|---|
dataset_id | str | Yes | Target dataset UUID. |
chunk_size | int | No | Byte chunk size to yield (default: 65536 bytes / 64 KB). |
timeout_ms | int | No | Request timeout in milliseconds. |