train.ConcatChunkedRandomSampler Class Reference

Inheritance diagram for train.ConcatChunkedRandomSampler:

Collaboration diagram for train.ConcatChunkedRandomSampler:

Public Member Functions
	__init__ (self, datasets, chunk_size=1_000_000)
	__iter__ (self)
	__len__ (self)

Public Attributes
	chunk_size = chunk_size
list	shard_lengths = [len(ds) for ds in datasets]
list	shard_offsets = []
list	chunk_descriptors = []
	num_samples = total

Detailed Description

Chunked shuffle that stays shard-local for ConcatDataset I/O locality.

Constructor & Destructor Documentation

__init__()

train.ConcatChunkedRandomSampler.__init__	(		self,
			datasets,
			chunk_size = 1_000_000 )

Member Function Documentation

__iter__()

train.ConcatChunkedRandomSampler.__iter__

(

self

)

__len__()

train.ConcatChunkedRandomSampler.__len__

(

self

)

Member Data Documentation

chunk_descriptors

list train.ConcatChunkedRandomSampler.chunk_descriptors = []

chunk_size

train.ConcatChunkedRandomSampler.chunk_size = chunk_size

num_samples

train.ConcatChunkedRandomSampler.num_samples = total

shard_lengths

train.ConcatChunkedRandomSampler.shard_lengths = [len(ds) for ds in datasets]

shard_offsets

list train.ConcatChunkedRandomSampler.shard_offsets = []

---

The documentation for this class was generated from the following file:

• training/train.py