U <c@sddlmZmZmZmZddlmZddlmZddl m Z ddl m Z mZmZmZdgZeddd Zed Gd ddeeZd S) )CallableIteratorOptionalTypeVar)functional_datapipe) IterDataPipe)dataframe_wrapper)_check_unpickable_fn_deprecation_warning StreamWrappervalidate_input_colFilterIterDataPipeT_coT) covariantfiltercs|eZdZUdZeed<eed<eed<deeeeddfdd Z ed d d Z e e d d d Z ddZddZZS)r a Filters out elements from the source datapipe according to input ``filter_fn`` (functional name: ``filter``). Args: datapipe: Iterable DataPipe being filtered filter_fn: Customized function mapping an element to a boolean. drop_empty_batches (Deprecated): By default, drops a batch if it is empty after filtering instead of keeping an empty list input_col: Index or indices of data which ``filter_fn`` is applied, such as: - ``None`` as default to apply ``filter_fn`` to the data directly. - Integer(s) is used for list/tuple. - Key(s) is used for dict. Example: >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> def is_even(n): ... return n % 2 == 0 >>> dp = IterableWrapper(range(5)) >>> filter_dp = dp.filter(filter_fn=is_even) >>> list(filter_dp) [0, 2, 4] datapipe filter_fndrop_empty_batchesN)rrrreturncs\t||_t|||_|dkr,d}ntt|jdddd||_||_ t ||dS)NTz1.12z1.14r)Zdeprecation_versionZremoval_versionZold_argument_name) super__init__rr rr type__name__r input_colr )selfrrrr __class__M/tmp/pip-unpacked-wheel-gikjz4vx/torch/utils/data/datapipes/iter/selecting.pyr0s zFilterIterDataPipe.__init__)rcsZ|jdkr|St|jttfrFtfdd|jD}|j|S||jSdS)Nc3s|]}|VqdSNr).0coldatarr Osz6FilterIterDataPipe._apply_filter_fn..)rr isinstancelisttuple)rr#argsrr"r_apply_filter_fnKs    z#FilterIterDataPipe._apply_filter_fnccs6|jD]*}||}||r&|Vqt|qdSr)r _returnIfTrue _isNonEmptyr Z close_streams)rr#filteredrrr__iter__Ts    zFilterIterDataPipe.__iter__cCs||}t|r\g}tt|D]\}}|r&|t||q&t|rXt|SdSt |t stt dt ||r||SdS)NzEBoolean output is required for `filter_fn` of FilterIterDataPipe, got) r) df_wrapperZ is_column enumerateZiterateappendZget_itemlenconcatr%bool ValueErrorr)rr# conditionresultidxmaskrrrr*\s    z FilterIterDataPipe._returnIfTruecCs8t|rdS|dk o2t|to0t|dko0|j }|S)NTr)r.Z is_dataframer%r&r1r)rr#rrrrr+os  zFilterIterDataPipe._isNonEmpty)NN)r __module__ __qualname____doc__r__annotations__rr3rrr)rrr-r*r+ __classcell__rrrrr s  N)typingrrrrZ%torch.utils.data.datapipes._decoratorrZ#torch.utils.data.datapipes.datapiperZ$torch.utils.data.datapipes.dataframerr.Z'torch.utils.data.datapipes.utils.commonr r r r __all__rr rrrrs