U &c@sddlZddlZddlmZddlmZmZmZmZddl Z ddl m Z dZ GdddZ Gdd d Z Gd d d ZGd d d ZGdddZGdddZGdddZdS)N) expanduser)DictListOptionalTuple)tqdmzhttps://huggingface.coc@s$eZdZdZeeeedddZdS)S3ObjzN Data structure that represents a file belonging to the current user. filename LastModifiedETagSizecKs||_||_||_||_dSNr )selfr r r r kwargsr7/tmp/pip-unpacked-wheel-ymerj3tt/transformers/hf_api.py__init__"szS3Obj.__init__N__name__ __module__ __qualname____doc__strintrrrrrrsrc@seZdZeeedddZdS) PresignedUrlwriteaccesstypecKs||_||_||_dSrr)rrrrrrrrr*szPresignedUrl.__init__N)rrrrrrrrrr)src@s&eZdZdZeeeeedddZdS)S3ObjectzL Data structure that represents a public file accessible on our S3. keyetag lastModifiedsize rfilenamecKs"||_||_||_||_||_dSrr!)rr"r#r$r%r&rrrrr5s zS3Object.__init__Nrrrrrr 0sr c@sBeZdZdZddggfeeeeeeeeeedddZ dS) ModelInfoz; Info about a public model accessible from our S3. NmodelIdr"author downloadstagssiblingscKs2||_||_||_||_||_dd|D|_dS)NcSsg|]}tf|qSr)r .0xrrr Ysz&ModelInfo.__init__..r()rr)r"r*r+r,r-rrrrrJs zModelInfo.__init__) rrrrrrrrrrrrrrr'Esr'c@seZdZdddZeeedddZeeeeefddd Zeddd d Z deee ee d d dZ deeee eedddZ dee eeedddZdeee edddZeedddZdS) HfApiNcCs|dk r |nt|_dSr)ENDPOINTendpoint)rr4rrrr]szHfApi.__init__)usernamepasswordreturncCs8d|j}tj|||dd}||}|dS)z Call HF API to sign in a user and get a token if credentials are valid. Outputs: token if credentials are valid Throws: requests.exceptions.HTTPError if credentials are invalid z {}/api/login)r5r6)jsontoken)formatr4requestspostraise_for_statusr8)rr5r6pathrdrrrlogin`s z HfApi.login)r9r7cCsDd|j}tj|dd|id}||}|d|dfS)z. Call HF API to know "whoami" z {}/api/whoami authorization Bearer {}headersuserZorgsr:r4r;getr=r8)rr9r>r?r@rrrwhoamips  z HfApi.whoamicCs0d|j}tj|dd|id}|dS)z) Call HF API to log out. z {}/api/logoutrBrCrDN)r:r4r;r<r=)rr9r>r?rrrlogoutzs z HfApi.logout)r9r organizationr7cCsFd|j}tj|dd|i||dd}||}tf|S)zP Call HF API to get a presigned url to upload `filename` to S3. z{}/api/presignrBrCr rKrEr8)r:r4r;r<r=r8r)rr9r rKr>r?r@rrrpresigns  z HfApi.presign)r9r filepathrKr7c Csn|j|||d}t|dH}t|}|jdkr2|nd}tj|j|d|jid} | | W5QRX|j S)z Get a presigned url, then upload file to S3. Outputs: url: Read-only url for the stored file on S3. rLrbrz content-type)datarE) rNopenTqdmProgressFileReader total_sizer;putrrr=closer) rr9r rOrKurlsfpfrRr?rrrpresign_and_uploads zHfApi.presign_and_upload)r9rKr7cCsXd|j}|dk rd|ind}tj||dd|id}||}dd|DS) z` Call HF API to list all stored files for user (or one of their organizations). z{}/api/listObjsNrKrBrC)paramsrEcSsg|]}tf|qSr)rr.rrrr1sz#HfApi.list_objs..rG)rr9rKr>r\r?r@rrr list_objss  zHfApi.list_objs)r9r rKcCs8d|j}tj|dd|i||dd}|dS)z= Call HF API to delete a file stored by user z{}/api/deleteObjrBrCrLrMN)r:r4r;deleter=)rr9r rKr>r?rrr delete_objs  zHfApi.delete_obj)r7cCs4d|j}t|}||}dd|DS)zf Get the public list of all the models on huggingface, including the community models z {}/api/modelscSsg|]}tf|qSr)r'r.rrrr1sz$HfApi.model_list..rG)rr>r?r@rrr model_lists   zHfApi.model_list)N)N)N)N)N)rrrrrrArrrIrJrrrNr[rr]r_r'r`rrrrr2\s    r2c@s2eZdZdZejdddZd ddZdd Zd S) rTu Wrap an io.BufferedReader `f` (such as the output of `open(…, "rb")`) and override `f.read()` so as to display a tqdm progress bar. see github.com/huggingface/transformers/pull/2078#discussion_r354739608 for implementation details. )rYcCs<||_t|j|_t|jdd|_|j|_|j |_dS)NF)totalZleave) rYosfstatfilenost_sizerUrpbarread_read)rrYrrrrs zTqdmProgressFileReader.__init__cCs|j|||Sr)rfupdaterg)rnrrrrhs zTqdmProgressFileReader._readcCs|jdSr)rfrW)rrrrrWszTqdmProgressFileReader.closeN)ri) rrrrioBufferedReaderrrhrWrrrrrTs rTc@s8eZdZedZeddZeddZeddZdS) HfFolderz~/.huggingface/tokenc Cs>tjtj|jddt|jd}||W5QRXdS)z8 Save token, creating folder as needed. T)exist_okzw+N)rbmakedirsr>dirname path_tokenrSr)clsr9rYrrr save_tokenszHfFolder.save_tokenc CsHz.t|jd}|W5QRWSQRXWntk rBYnXdS)z4 Get token or None if not existent. r?N)rSrrrgFileNotFoundError)rsrYrrr get_tokens  zHfFolder.get_tokencCs*zt|jWntk r$YnXdS)zL Delete token. Do not fail if token does not exist. N)rbremoverrru)rsrrr delete_tokenszHfFolder.delete_tokenN) rrrrrr classmethodrtrvrxrrrrrns  rn)rlrbos.pathrtypingrrrrr;rr3rrr r'r2rTrnrrrrs   l