|Yh#yvddlZddlZddlZddlZddlZddlZddlmcmcm Z ddl mZddl m Z ddlmZddlmZmZmZmZmZmZddlmZmZmZmZmZmZmZmZddl m!Z!m"Z"ddl#m$Z$dd l%m&Z&m'Z'm(Z(gd Z)eZ*ej+ej,j+ej-ej,j-iej,j+ej j+ej,j-ej j-id Z.d Z/ d#dZ0d$dZ1dZ2dZ3d%dZ4 d&dZ5dZ6dZ7ej8e& d'dZ9dZ:dZ;ej8e&d(dZ<ej8e&dej=ddfdZ>ej8e&d(dZ?ej8e&d%dZ@ej8e& d)dZA d*d ZB d%d!ZCd+d"ZDdS),N) _FusedModule)_is_activation_post_process)_activation_is_memoryless_add_module_to_qconfig_obs_ctrdefault_dynamic_qconfigfloat16_dynamic_qconfig!float_qparams_weight_only_qconfig&float_qparams_weight_only_qconfig_4bit)_get_special_act_post_process_has_special_act_post_process)get_default_dynamic_quant_module_mappingsget_default_qat_module_mappings$get_default_qconfig_propagation_list(get_default_static_quant_module_mappings2get_default_static_quant_reference_module_mappingsno_observer_set) DeQuantStub QuantWrapper)type_before_parametrizations)DEPRECATION_WARNINGget_qparam_dict)has_no_children_ignoring_parametrizations) get_default_custom_config_dictpropagate_qconfig_add_quant_dequantpreparequantizequantize_dynamic prepare_qat quantize_qatconvert swap_module)%float_to_observed_custom_module_class)observed_to_quantized_custom_module_classctS)z'Defines the default custom config dict.)_DEFAULT_CUSTOM_CONFIG_DICTp/var/www/tools.fuzzalab.pt/emblema-extractor/venv/lib/python3.11/site-packages/torch/ao/quantization/quantize.pyrrGs &&r)c |t||}|||}t|d|}tjjj||t||}||_| D]b\}}|r|dz|zn|} |=||dgvs7t||dgvst|||| cdS)aThis is a helper function for `propagate_qconfig_` Args: module: input module qconfig_dict: dictionary that maps from name of submodule to quantization configuration qconfig_parent: quantization config of parent module, we will fallback to this config when there is no specified config for current module prefix: corresponding prefix of the current module, used as key in qconfig_dict prepare_custom_config_dict: dictionary for custom handling of modules see docs for :func:`~torch.ao.quantization.prepare_fx` Return: None, module is modified inplace with qconfig attached qconfig.Nnon_traceable_module_namenon_traceable_module_class) getrgetattrtorchao quantizationr-_assert_valid_qconfigrnamed_childrentype_propagate_qconfig_helper) module qconfig_dictqconfig_parentprefixprepare_custom_config_dictmodule_qconfigqconfig_with_device_checknamechild module_prefixs r*r9r9Ls02"%%$V,,nN"%%fn==NVY??N H!77OOO >~v V V.FN,,..   e/5? t++4 % - .223NPRSS S SE{{)--.JBOOPP &|%>      r)c:|i}|i}t|||dS)aPropagate qconfig through the module hierarchy and assign `qconfig` attribute on each leaf module Args: module: input module qconfig_dict: dictionary that maps from name or type of submodule to quantization configuration, qconfig applies to all submodules of a given module unless qconfig for the submodules are specified (when the submodule already has qconfig attribute) prepare_custom_config_dict: dictionary for custom handling of modules see docs for :func:`~torch.ao.quantization.prepare_fx` Return: None, module is modified inplace with qconfig attached N)r>)r9)r:r;r>s r*rr}sD  !)%'" 9Sr)c,||S)z.Forward hook that calls observer on the outputactivation_post_process)selfinputoutputs r*_observer_forward_hookrKs  ' ' / //r)c8||dS)z2Forward pre hook that calls observer on the outputrrF)rHrIs r*_observer_forward_pre_hookrMs  ' 'a 1 11r)Fct|ds Jd|r|tddS|tddS)NrGzGExpect activation_post_process attribute already attached to the moduleT)prepend)hasattrregister_forward_pre_hookrMregister_forward_hookrK)r:pre_hooks r*&_register_activation_post_process_hookrTsw 64 5 5QK(()CT(RRRRR$$%;T$JJJJJr)c |t}|i}`t|}t|dks Jd|t|dkrtt |ndd d d d  fd }|D]\}}t |tjfvr#tt |tj tj frM |rAt|dsJd t |d  |j|_t!|t"r |r |||(t ||vr |r ||t%|rt'|} ||| % |rt ||vrq|t |} | |} t+||| t| t-t/s || t1||||t3|r;t!|t4jjst ||vr ||t|d r=t!|t4jjs t ||vr||dSdSdSdS) asAdd observer for the leaf child of the module. This function insert observer module to all leaf child module that has a valid qconfig attribute. Args: module: input module with qconfig attributes for all the leaf modules that we want to quantize qconfig_propagation_list: a list of quantizable modules that will have observers added to them if they are leaf nodes device: parent device, if any non_leaf_module_list: list of non-leaf modules we want to add observer Return: None, module is modified inplace with added observer modules and forward_hooks NrzR_add_observer_ only works with cpu or single-device CUDA modules, but got devices rct||n |}||||SN) activationto)r-devicespecial_act_post_processrXs r*get_activation_post_processz3_add_observer_..get_activation_post_processsO(/    ))++    MM& ! ! !r)c4t|do|jduS)Nr-rPr-)ms r*needs_observationz)_add_observer_..needs_observationsq)$$>$)>>r)c|r`t|tsM|d|j|t |t |jdSdSdS)zmAdds an activation post process module and register a pre or post hook that calls the module rGrSN) isinstancer add_moduler-rTr)r_r[rZr\r`s r*insert_activation_post_processz6_add_observer_..insert_activation_post_processs  Q    1k(B(B  LL)++Iv'?    35ai@@          r)rGzfunctional class z- has no pre-defined `activation_post_process`weight_fake_quantrW)r_get_unique_devices_lennextiterr7rnnDropout issubclassnnqFloatFunctional QFunctionalrPr-rGrcrr r from_floatsetattrtupler_add_observer_rr3 Sequential)r:qconfig_propagation_listnon_leaf_module_listrZcustom_module_class_mappingdevicesrerArBr[observed_classobserved_childr\r`s ` @@r*rtrts, '#G#I#I "*&(#~&v..7||q   jah j j!  ),G q(8(8d7mm$$$d???&,,..// e ' . .2:, > >   ( / /#2Es1W  + ! '' u&?@@z(DU(K(Kzzz1L0KM611-| , ,!   '' 6..u555 ,,U337KKK  '' 6..u555 *5 1 1 'DU'K'K $ * *52J K K K K  e $ $ ,U337RRR8,U33N,66u==N FD. 1 1 1neO4E4E.F.FGG ?..~>>> ($+      2&99/658#677/ ) 0 04L L L&&v... +,,/658#677/ ) 0 04L L L&&v..... //// M Lr)c|d|Dd|DzS)Nc<h|]}|jjdk|jSmetarZr8.0ps r* z'_get_unique_devices_..4s( M M MQX]f5L5LAH5L5L5Lr)c<h|]}|jjdk|jSr~rrs r*rz'_get_unique_devices_..4s2QQQahmv.E.E.E.E.Er)) parametersbuffers)r:s r*rgrg3sT M Mf//11 M M MQQ ..**QQQ r)ct|r&t|dr|jrt|S|D]\}}t ||j|<|S)a{Wrap the leaf child module in QuantWrapper if it has a valid qconfig Note that this function will modify the children of module inplace and it can return a new module which wraps the input module as well. Args: module: input module with qconfig attributes for all the leaf modules that we want to quantize Return: Either the inplace modified module with submodules wrapped in `QuantWrapper` based on qconfig or a new `QuantWrapper` module which wraps the input module, the latter case only happens when the input module is a leaf module and we want to quantize it. r-)rrPr-rr7r_modules)r:rArBs r*rr9s} 2&99$ FI & &$ N$ F###,,..99 e 1% 8 8 Mr)ctjd|t}|di}|st j|}|}|t}t|dtd| Dstj dt|||||S)afPrepares a copy of the model for quantization calibration or quantization-aware training. Quantization configuration should be assigned preemptively to individual submodules in `.qconfig` attribute. The model will be attached with observer or fake quant modules, and qconfig will be propagated. Args: `model`: input model to be modified in-place `inplace`: carry out model transformations in-place, the original module is mutated `allow_list`: list of quantizable modules `observer_non_leaf_module_list`: list of non-leaf modules we want to add observer `prepare_custom_config_dict`: customization configuration dictionary for prepare function .. code-block:: python # Example of prepare_custom_config_dict: prepare_custom_config_dict = { # user will manually define the corresponding observed # module class which has a from_float class method that converts # float custom module to observed custom module "float_to_observed_custom_module_class": {CustomModule: ObservedCustomModule} } z!quantization_api.quantize.prepareNr$r;c3DK|]}t|do|jVdS)r-Nr^)rr_s r* zprepare..s3LLqwq)$$2LLLLLLr)zNone of the submodule got qconfig applied. Make sure you passed correct configuration through `qconfig_dict` or by assigning the `.qconfig` attribute directly on submodules)rx)r3_C_log_api_usage_oncerr1copydeepcopyrranymoduleswarningswarnrt)modelinplace allow_listobserver_non_leaf_module_listr>rxrvs r*rrTsD H  !DEEE!)%C%E%E""<"@"@/## % e$$ *#G#I#I u40000 LLEMMOOLLL L L   K     %$?  Lr)ctdr$tjrtddfd }|d|ddS)NrGFc|rjnj}|rtnt}t }|D]\}}||ur|||D]}||dSrW)_forward_pre_hooks_forward_hooksrMrKsetitemsaddpop)rShook_map observer_hookhandle_ids_to_remove handle_idhook_fnr:s r* remove_hooksz5_remove_activation_post_process..remove_hookss08S6,,f>S*2 N & &8N  #uu"*.."2"2 4 4 Iw-''$((333- $ $I LL # # # # $ $r)TrbF)rPrrGdelattr)r:rs` r*_remove_activation_post_processrsv01136Q&773 1222 $ $ $ $ $ $L$L%      r)c|D]}t|t|dr|`t |dS)zClean up the qconfig left in the module so that new qconfig can be propagated. Args: module: module to be cleaned up r-N)children_remove_qconfigrPr-r)r:rBs r*rrsY""vy!! N#F+++++r)ctjd|t}|st j|}|t|d||g|Rt||d|S)aQuantize the input float model with post training static quantization. First it will prepare the model for calibration, then it calls `run_fn` which will run the calibration step, after that we will convert the model to a quantized model. Args: model: input float model run_fn: a calibration function for calibrating the prepared model run_args: positional arguments for `run_fn` inplace: carry out model transformations in-place, the original module is mutated mapping: correspondence between original module types and quantized counterparts Return: Quantized model. z"quantization_api.quantize.quantizeNTr) r3rrrrrevalrr")rrun_fnrun_argsmappingrs r*rrs$ H  !EFFF:<< % e$$ JJLLL E4     F58 E7D)))) Lr)c 4tjd|`|tjkrjtjt tjt tjt tj t tj t tj t i}n|tj krjtjttjttjttj ttj ttj ti}n%|tjkr%tjt tjt i}n|tjkrtjt&i}nt)d|dt+|t,r|tjurt }n_|tj urt}nI|tjurt }n3|tjurt&}nt/dt1|t3t5|t7j|}|t;}|st=j|}| tC||tE||d|S)avConverts a float model to dynamic (i.e. weights-only) quantized model. Replaces specified modules with dynamic weight-only quantized versions and output the quantized model. For simplest usage provide `dtype` argument that can be float16 or qint8. Weight-only quantization by default is performed for layers with large weights size - i.e. Linear and RNN variants. Fine grained control is possible with `qconfig` and `mapping` that act similarly to `quantize()`. If `qconfig` is provided, the `dtype` argument is ignored. Args: model: input model qconfig_spec: Either: - A dictionary that maps from name or type of submodule to quantization configuration, qconfig applies to all submodules of a given module unless qconfig for the submodules are specified (when the submodule already has qconfig attribute). Entries in the dictionary need to be QConfig instances. - A set of types and/or submodule names to apply dynamic quantization to, in which case the `dtype` argument is used to specify the bit-width inplace: carry out model transformations in-place, the original module is mutated mapping: maps type of a submodule to a type of corresponding dynamically quantized version with which the submodule needs to be replaced z*quantization_api.quantize.quantize_dynamicNz5Don't know how to quantize with default settings for z. Provide full qconfig pleasez.Unknown dtype specified for quantize_dynamic: Tr)#r3rrqint8rkLinearrLSTMGRULSTMCellRNNCellGRUCellfloat16rquint8 EmbeddingBagr Embeddingquint4x2r ValueErrorrcr RuntimeErrorstrdictzip itertoolsrepeatr rrrrr")r qconfig_specdtyperrdefault_qconfigs r*rrs@ H  !MNNN EK   20/ 4 3 3 LLem # # 20/ 4 3 3 LLel " "!B ?LLen $ $!GLLllll  L# & & R EK  5OO em # #5OO el " "?OO en $ $DOO@#e** C i.>.O.OPPQQ ;== % e$$ JJLLLul+++ E7D)))) Lr)c\tjd|js Jd|t }|st j|}t|dt||ddt|t| d|S) a Prepares a copy of the model for quantization calibration or quantization-aware training and converts it to quantized version. Quantization configuration should be assigned preemptively to individual submodules in `.qconfig` attribute. Args: model: input model to be modified in-place mapping: dictionary that maps float modules to quantized modules to be replaced. inplace: carry out model transformations in-place, the original module is mutated z%quantization_api.quantize.prepare_qatz1prepare_qat only works on models in training modeNrTF)rrremove_qconfig)rr) r3rrtrainingrrrrr"rrvalues)rrrs r*r r :s  H  !HIII >NNNNNN133 % e$$u40000 E7DGGGG EW^^5E5E1F1FPTUUUU Lr)ctjd|stj|}|t |d||g|Rt|d|S)agDo quantization aware training and output a quantized model Args: model: input model run_fn: a function for evaluating the prepared model, can be a function that simply runs the prepared model or a training loop run_args: positional arguments for `run_fn` Return: Quantized model. z&quantization_api.quantize.quantize_qatTr)r3rrrrtrainr r")rrrrs r*r!r!Xs H  !IJJJ % e$$ KKMMMt$$$$ F58 E4    Lr)Tctjd|stj|}t ||d||||rt ||S)aConverts submodules in input module to a different module according to `mapping` by calling `from_float` method on the target module class. And remove qconfig at the end if remove_qconfig is set to True. Args: `module`: prepared and calibrated module `mapping`: a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules `inplace`: carry out model transformations in-place, the original module is mutated `convert_custom_config_dict`: custom configuration dictionary for convert function `use_precomputed_fake_quant`: a flag to enable use of precomputed fake quant .. code-block:: python # Example of convert_custom_config_dict: convert_custom_config_dict = { # user will manually define the corresponding quantized # module class which has a from_observed class method that converts # observed custom module to quantized custom module "observed_to_quantized_custom_module_class": { ObservedCustomModule: QuantizedCustomModule } } z!quantization_api.quantize.convertT)r is_referenceconvert_custom_config_dictuse_precomputed_fake_quant)r3rrrr_convertr)r:rrrrrrs r*r"r"psyJ H  !DEEE 'v&& !#=#=   Mr)c ||rtn t}|t}|di}|st j|}i}|D]U\}} t| ts&t| |vrt| |d|||t| |||||<V| D]\} } | |j | <|S)aoConverts submodules in input module to a different module according to `mapping` by calling `from_float` method on the target module class Args: module: input module mapping: a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules inplace: carry out model transformations in-place, the original module is mutated is_reference: a flag to enable quantized reference module use_precomputed_fake_quant: a flag to enable use of precomputed fake quant Nr%Tr)rrrr1rrr7rcrrrr#rr) r:rrrrrrxreassignrAmodkeyvalues r*rrs?, < > @ @ @9;;  ")%C%E%E""<"@"@3R## 'v&&H**,,  c3 -- ,S119TTT *+E     % 57Q  nn&&%% U$ Mr)c|}t|dr:|j2d}t||vr+|t||}d}nt||vr|t|}t|dr_|jrX|jJ|j}||jt |}|||}nOtj |j} d| j vr|||}n||}d}|r |j D]} | | |j D] } | tur|| !t#|} t%| dks6t%| d krt'jd | vs Jd | t%| d krt+t-| nd} | r|| |S) a Swaps the module if it has a quantized counterpart and it has an `observer` attached. Args: mod: input module mapping: a dictionary that maps from nn module to nnq module Return: The corresponding quantized module of `mod` r-NFT _IS_REFERENCErrrrzOswap_module only works with cpu or single-device CUDA modules, but got devices r)rPr-r from_observedrweightrrqinspect signaturerrrrQrrKrRrgrhr3rZrirjrY)rrrxrnew_modswappedqmodweight_post_processweight_qparamssig pre_hook_fnrryrZs r*r#r#sGsI,#3;#: ' , ,0K K K1,S11mC   GG )# . .' 9 97<<=Dt_-- 3$2D 3{...&)k&8&8&:&:###CJ///!01D!E!E//#~>>'88/3>AA"oo8R.GG#ooc22GG  #"5<<>> ? ? 11+>>>>-4466 ; ;"88811':::+3//Gw<<1$$G !!el6&:&:g&E&E&Ekbikk'F&E&E-0LL1,<,.get_prefix*s2vv6C<7r)rGN)rPrGr7_get_observer_dict)r target_dictr=rrArBrCs r*rr!s888s-..  ' JJv&&)BBC))++>> e5;E 6**T11 5+}====>>r))Nr+N)NNr)NNNN)FNNN)NF)NFTFNF)NFFNF)r+)Errrtyping_extensionsrr3torch.ao.nn.quantizedr4rk quantizedrntorch.nntorch.ao.nn.intrinsicrtorch.ao.quantization.observerrtorch.ao.quantization.qconfigrrrrr r +torch.ao.quantization.quantization_mappingsr r r rrrrrtorch.ao.quantization.stubsrrtorch.nn.utils.parametrizerutilsrrr__all__is_activation_post_processr quantizableMultiheadAttentionr'rr9rrKrMrTrtrgr deprecatedrrrrrrr r!r"rr#rr(r)r*rs  ############......FFFFFF                    BAAAAAAACCCCCC   9 $ r~@. R\. )2<+J2  ''' # ....b2000 222 KKKK"  $ F/F/F/F/R 6122 "&# ???32?D!!!4 , , , 12232:122EKuWWW32Wt12232:12232.122  #$111321l  #$ ;;;;~KP;;;;|>>>>>>r)