Dataset

Dataset is a storage for managing structured tabular data.

The dataset class provides a high-level interface for storing and retrieving structured data with consistent schema, similar to database tables or spreadsheets. It abstracts the underlying storage implementation details, offering a consistent API regardless of where the data is physically stored.

Dataset operates in an append-only mode, allowing new records to be added but not modified or deleted after creation. This makes it particularly suitable for storing crawling results and other data that should be immutable once collected.

The class provides methods for adding data, retrieving data with various filtering options, and exporting data to different formats. You can create a dataset using the open class method, specifying either a name or ID. The underlying storage implementation is determined by the configured storage client.

Usage

from crawlee.storages import Dataset

# Open a dataset
dataset = await Dataset.open(name='my_dataset')

# Add data
await dataset.push_data({'title': 'Example Product', 'price': 99.99})

# Retrieve filtered data
results = await dataset.get_data(limit=10, desc=True)

# Export data
await dataset.export_to('results.json', content_type='json')

Hierarchy

• Storage
  • Dataset

Index

Methods

• __init__
• drop
• export_to
• get_data
• get_metadata
• iterate_items
• list_items
• open
• purge
• push_data

Properties

• id
• name

Methods

__init__

• __init__(client, id, name): None

• Initialize a new instance.

  Preferably use the Dataset.open constructor to create a new instance.

  ---

  Parameters

  • client: DatasetClient

    An instance of a storage client.

  • id: str

    The unique identifier of the storage.

  • name: str | None

    The name of the storage, if available.

  Returns None

drop

• async drop(): None

• Overrides RequestManager.drop

  Drop the storage, removing it from the underlying storage client and clearing the cache.

  ---

  Returns None

export_to

• async export_to(key: str, content_type?: Literal[json, csv], to_kvs_id?: str | None, to_kvs_name?: str | None, to_kvs_storage_client?: StorageClient | None, to_kvs_configuration?: Configuration | None, kwargs: Any): None
• async export_to(key: str, content_type: Literal[json], to_kvs_id?: str | None, to_kvs_name?: str | None, to_kvs_storage_client?: StorageClient | None, to_kvs_configuration?: Configuration | None, *: , skipkeys: NotRequired[bool], ensure_ascii: NotRequired[bool], check_circular: NotRequired[bool], allow_nan: NotRequired[bool], cls: NotRequired[type[json.JSONEncoder]], indent: NotRequired[int], separators: NotRequired[tuple[str, str]], default: NotRequired[Callable], sort_keys: NotRequired[bool]): None
• async export_to(key: str, content_type: Literal[csv], to_kvs_id?: str | None, to_kvs_name?: str | None, to_kvs_storage_client?: StorageClient | None, to_kvs_configuration?: Configuration | None, *: , dialect: NotRequired[str], delimiter: NotRequired[str], doublequote: NotRequired[bool], escapechar: NotRequired[str], lineterminator: NotRequired[str], quotechar: NotRequired[str], quoting: NotRequired[int], skipinitialspace: NotRequired[bool], strict: NotRequired[bool]): None

• Export the entire dataset into a specified file stored under a key in a key-value store.

  This method consolidates all entries from a specified dataset into one file, which is then saved under a given key in a key-value store. The format of the exported file is determined by the content_type parameter. Either the dataset's ID or name should be specified, and similarly, either the target key-value store's ID or name should be used.

  ---

  Parameters

  • key: str

    The key under which to save the data in the key-value store.

  • optionalcontent_type: Literal[json, csv] = 'json'

    The format in which to export the data.

  • optionalto_kvs_id: str | None = None

    ID of the key-value store to save the exported file. Specify only one of ID or name.

  • optionalto_kvs_name: str | None = None

    Name of the key-value store to save the exported file. Specify only one of ID or name.

  • optionalto_kvs_storage_client: StorageClient | None = None

    Storage client to use for the key-value store.

  • optionalto_kvs_configuration: Configuration | None = None

    Configuration for the key-value store.

  • kwargs: Any

    Additional parameters for the export operation, specific to the chosen content type.

  Returns None

get_data

• async get_data(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden, flatten, view): DatasetItemsListPage

• Retrieve a paginated list of items from a dataset based on various filtering parameters.

  This method provides the flexibility to filter, sort, and modify the appearance of dataset items when listed. Each parameter modifies the result set according to its purpose. The method also supports pagination through 'offset' and 'limit' parameters.

  ---

  Parameters

  • optionalkeyword-onlyoffset: int = 0

    Skips the specified number of items at the start.

  • optionalkeyword-onlylimit: int | None = 999_999_999_999

    The maximum number of items to retrieve. Unlimited if None.

  • optionalkeyword-onlyclean: bool = False

    Return only non-empty items and excludes hidden fields. Shortcut for skip_hidden and skip_empty.

  • optionalkeyword-onlydesc: bool = False

    Set to True to sort results in descending order.

  • optionalkeyword-onlyfields: list[str] | None = None

    Fields to include in each item. Sorts fields as specified if provided.

  • optionalkeyword-onlyomit: list[str] | None = None

    Fields to exclude from each item.

  • optionalkeyword-onlyunwind: str | None = None

    Unwinds items by a specified array field, turning each element into a separate item.

  • optionalkeyword-onlyskip_empty: bool = False

    Excludes empty items from the results if True.

  • optionalkeyword-onlyskip_hidden: bool = False

    Excludes fields starting with '#' if True.

  • optionalkeyword-onlyflatten: list[str] | None = None

    Fields to be flattened in returned items.

  • optionalkeyword-onlyview: str | None = None

    Specifies the dataset view to be used.

  Returns DatasetItemsListPage

get_metadata

• async get_metadata(): (DatasetMetadata | KeyValueStoreMetadata) | RequestQueueMetadata

• Overrides Storage.get_metadata

  Get the storage metadata.

  ---

  Returns (DatasetMetadata | KeyValueStoreMetadata) | RequestQueueMetadata

iterate_items

• async iterate_items(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden): AsyncIterator[dict[str, Any]]

• Iterate over items in the dataset according to specified filters and sorting.

  This method allows for asynchronously iterating through dataset items while applying various filters such as skipping empty items, hiding specific fields, and sorting. It supports pagination via offset and limit parameters, and can modify the appearance of dataset items using fields, omit, unwind, skip_empty, and skip_hidden parameters.

  ---

  Parameters

  • optionalkeyword-onlyoffset: int = 0

    Skips the specified number of items at the start.

  • optionalkeyword-onlylimit: int | None = 999_999_999_999

    The maximum number of items to retrieve. Unlimited if None.

  • optionalkeyword-onlyclean: bool = False

    Return only non-empty items and excludes hidden fields. Shortcut for skip_hidden and skip_empty.

  • optionalkeyword-onlydesc: bool = False

    Set to True to sort results in descending order.

  • optionalkeyword-onlyfields: list[str] | None = None

    Fields to include in each item. Sorts fields as specified if provided.

  • optionalkeyword-onlyomit: list[str] | None = None

    Fields to exclude from each item.

  • optionalkeyword-onlyunwind: str | None = None

    Unwinds items by a specified array field, turning each element into a separate item.

  • optionalkeyword-onlyskip_empty: bool = False

    Excludes empty items from the results if True.

  • optionalkeyword-onlyskip_hidden: bool = False

    Excludes fields starting with '#' if True.

  Returns AsyncIterator[dict[str, Any]]

list_items

• async list_items(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden): list[dict[str, Any]]

• Retrieve a list of all items from the dataset according to specified filters and sorting.

  This method collects all dataset items into a list while applying various filters such as skipping empty items, hiding specific fields, and sorting. It supports pagination via offset and limit parameters, and can modify the appearance of dataset items using fields, omit, unwind, skip_empty, and skip_hidden parameters.

  ---

  Parameters

  • optionalkeyword-onlyoffset: int = 0

    Skips the specified number of items at the start.

  • optionalkeyword-onlylimit: int | None = 999_999_999_999

    The maximum number of items to retrieve. Unlimited if None.

  • optionalkeyword-onlyclean: bool = False

    Return only non-empty items and excludes hidden fields. Shortcut for skip_hidden and skip_empty.

  • optionalkeyword-onlydesc: bool = False

    Set to True to sort results in descending order.

  • optionalkeyword-onlyfields: list[str] | None = None

    Fields to include in each item. Sorts fields as specified if provided.

  • optionalkeyword-onlyomit: list[str] | None = None

    Fields to exclude from each item.

  • optionalkeyword-onlyunwind: str | None = None

    Unwinds items by a specified array field, turning each element into a separate item.

  • optionalkeyword-onlyskip_empty: bool = False

    Excludes empty items from the results if True.

  • optionalkeyword-onlyskip_hidden: bool = False

    Excludes fields starting with '#' if True.

  Returns list[dict[str, Any]]

open

• async open(*, id, name, configuration, storage_client): Storage

• Overrides Storage.open

  Open a storage, either restore existing or create a new one.

  ---

  Parameters

  • optionalkeyword-onlyid: str | None = None

    The storage ID.

  • optionalkeyword-onlyname: str | None = None

    The storage name.

  • optionalkeyword-onlyconfiguration: Configuration | None = None

    Configuration object used during the storage creation or restoration process.

  • optionalkeyword-onlystorage_client: StorageClient | None = None

    Underlying storage client to use. If not provided, the default global storage client from the service locator will be used.

  Returns Storage

purge

• async purge(): None

• Overrides Storage.purge

  Purge the storage, removing all items from the underlying storage client.

  This method does not remove the storage itself, e.g. don't remove the metadata, but clears all items within it.

  ---

  Returns None

push_data

• async push_data(data): None

• Store an object or an array of objects to the dataset.

  The size of the data is limited by the receiving API and therefore push_data() will only allow objects whose JSON representation is smaller than 9MB. When an array is passed, none of the included objects may be larger than 9MB, but the array itself may be of any size.

  ---

  Parameters

  • data: list[dict[str, Any]] | dict[str, Any]

    A JSON serializable data structure to be stored in the dataset. The JSON representation of each item must be smaller than 9MB.

  Returns None

Properties

id

id: str

Overrides Storage.id

Get the storage ID.

name

name: str | None

Overrides Storage.name

Get the storage name.