OmniFold: A Method to Simultaneously Unfold All Observables

MIT-CTP 5155 OmniF old: A Metho d to Sim ultaneously Unfold All Observ ables Anders Andreassen, 1, 2, 3 , ∗ P atrick T. Komisk e, 4 , † Eric M. Meto diev, 4 , ‡ Benjamin Nachman, 2 , § and Jesse Thaler 4 , ¶ 1 Dep artment of Physics, University of California, Berkeley, CA 94720, USA 2 Physics Division, L awr enc e Berkeley National L ab or atory, Berkeley, CA 94720, USA 3 Go o gle, Mountain View, CA 94043, USA 4 Center for The or etic al Physics, Massachusetts Institute of T echnolo gy, Cambridge, MA 02139, USA Collider data must be corrected for detector effects (“unfolded”) to b e compared with many theoretical calculations and measurements from other experiments. Unfolding is traditionally done for individual, binned observ ables without including all information relev an t for characterizing the detector resp onse. W e in tro duce OmniF old , an unfolding metho d that iteratively reweigh ts a sim ulated dataset, using machine learning to capitalize on all av ailable information. Our approach is un binned, works for arbitrarily high-dimensional data, and naturally incorp orates information from the full phase space. W e illustrate this tec hnique on a realistic jet substructure example from the Large Hadron Collider and compare it to standard binned unfolding metho ds. This new paradigm enables the simultaneous measuremen t of all observ ables, including those not y et in ven ted at the time of the analysis. Measuring properties of particle collisions is a cen tral goal of particle physics exp erimen ts, suc h as those at the Large Hadron Collider (LHC). After correcting for detec- tor effects, distributions of collider observ ables at “truth lev el” can b e compared with semi-inclusive theoretical predictions as well as with measurements from other ex- p erimen ts. These comparisons are widely used to en- hance our understanding of the Standard Mo del, tune parameters of Monte Carlo even t generators, and enable precision searches for new ph ysics. “Unfolding” is the pro cess of obtaining these truth distributions (particle- lev el) from measured information recorded by a detector (detector-lev el). The unfolding process ensures that mea- suremen ts are indep enden t of the specific experimental con text, allowing for comparisons across different exp er- imen ts and usage with the latest theoretical tools, 1 ev en long after the original analysis is completed. Many un- folding methods hav e been prop osed and are currently used by experiments. See Refs. [ 1 – 4 ] for reviews and Refs. [ 5 – 7 ] for the most widely-used unfolding algorithms. Curren t unfolding metho ds face three key challenges. First, all of the widely-used metho ds require the mea- sured observ ables to b e binned into histograms. This binning must b e determined ahead of time and is of- ten chosen manually . Second, b ecause the measurements are binned, one can only unfold a small num b er of ob- serv ables sim ultaneously . Multi-differen tial cross section measuremen ts b ey ond t wo or three dimensions are simply not feasible. Finally , unfolding corrections for detector effects often do not take into account all p ossible aux- iliary features that control the detector resp onse. Even though the inputs to the unfolding can b e calibrated, if the detector resp onse dep ends on features that are not used directly in the unfolding, then the results will b e 1 F or fully exclusive theoretical predictions, one could alternatively forward fold to compare to exp erimen tal data. sub optimal and p oten tially biased. This letter introduces OmniF old , a new approach that solves all three of these unfolding challenges. Detector-lev el quan tities are iteratively unfolded, using mac hine learning to handle phase space of any dimension- alit y without requiring binning. Utilizing the full phase space information mitigates the problem of auxiliary fea- tures con trolling the detector resp onse. There hav e b een previous proposals to use machine learning metho ds for unfolding [ 8 – 10 ] as w ell as proposals to perform unfolding without binning [ 9 – 12 ]. These proposals, how ev er, are un tenable in high dimensions and do not reduce to stan- dard methods in the binned case. OmniFold naturally pro cesses high-dimensional features, in the spirit of pre- vious mac hine-learning-based reweigh ting strategies [ 13 – 18 ], and it reduces to well-established methods [ 5 ] in the binned case. W e also in tro duce simpler versions of the pro cedure, using single or multiple observ ables, named UniF old and Mul tiFold , resp ectiv ely . 2 All unfolding metho ds require a trustable detector sim ulation to estimate the detector resp onse. In the binned formulation, the folding equation can b e written as m = R t , where m and t are vectors of the measured detector-lev el and true particle-level histograms, resp ec- tiv ely . R is the “resp onse matrix”: R ij = Pr(measure i | truth is j ) . (1) In general, R is not inv ertible, so the unfolding problem has no unique solution, and metho ds attempt to achiev e a useful solution in v arious w ays. One of the most widely- used metho ds is Iterative Bay esian Unfolding (IBU) [ 5 ], also kno wn as Richardson-Lucy deconv olution [ 20 , 21 ]. Giv en a measured sp ectrum m i = Pr(measure i ) and a 2 The name OmniF old is tak en from Emily Dic kinson’s p oem The Mountain Sat Up on the Plain [ 19 ]. 2 prior sp ectrum t (0) j = Pr 0 (truth is j ), IBU proceeds iter- ativ ely according to the equation: t ( n ) j = X i Pr n − 1 (truth is j | measure i ) Pr(measure i ) = X i R ij t ( n − 1) j P k R ik t ( n − 1) k × m i , (2) where n is the iteration n umber. OmniF old uses mac hine learning to generalize Eq. ( 2 ) to the unbinned, full phase space. A key concept for this approac h is the likelihoo d ratio: L [( w , X ) , ( w 0 , X 0 )]( x ) = p ( w,X ) ( x ) p ( w 0 ,X 0 ) ( x ) , (3) where p ( w,X ) is the probability densit y of x estimated from empirical weigh ts w and samples X . The function L [( w , X ) , ( w 0 , X 0 )]( x ) can b e approx imated using a clas- sifier trained to distinguish ( w , X ) from ( w 0 , X 0 ). This prop ert y has been successfully exploited using neural net- w orks for full phase-space Monte Carlo reweigh ting and parameter estimation [ 18 , 22 – 26 ]. Here, we use neural net work classifiers to iteratively reweigh t the particle- and detector-level Monte Carlo w eights, resulting in an unfolding procedure. The OmniFold technique is illustrated in Fig. 1 . In- tuitiv ely , synthetic detector-level even ts (“sim ulation”) are rew eigh ted to matc h experimental data (“data”), and then the rew eighted synthetic ev en ts, no w ev aluated at particle-lev el (“generation”), are further rew eighted to estimate the true particle-level information (“truth”). The starting p oin t is a synthetic Monte Carlo dataset comp osed of pairs ( t, m ), where eac h particle-level even t t is pushed through the detector simulation to obtain a detector-lev el even t m . Particle-lev el even ts hav e initial w eights ν 0 ( t ), and when t is pushed to m , these become detector-lev el w eigh ts ν push 0 ( m ) = ν 0 ( t ). OmniFold it- erates the following steps: 1. ω n ( m ) = ν push n − 1 ( m ) L [(1 , Data) , ( ν push n − 1 , Sim.)]( m ), 2. ν n ( t ) = ν n − 1 ( t ) L [( ω pull n , Gen.) , ( ν n − 1 , Gen.)]( t ) . The first step yields new detector-level weigh ts ω n ( m ), whic h are pulled back to particle-level weigh ts ω pull n ( t ) = ω n ( m ) using the same synthetic pairs ( t, m ). Note that ν push and ω pull are not, strictly sp eaking, functions b e- cause of the m ulti-v alued nature of the detector simula- tion. The second step ensures that ν n is a v alid w eigh ting function of the particle-level quantities. Assuming ν 0 ( t ) = 1, in the first iteration Step 1 learns ω 1 ( m ) = p Data ( m ) /p Sim. ( m ), which is pulled bac k to the particle-lev el w eights ω pull 1 ( t ). Step 2 simply conv erts the p er-instance weigh ts ω pull 1 ( t ) to a v alid particle-lev el w eighting function ν 1 ( t ). After one iteration, the new Simulation Synthetic Natural Detector-lev el Data P ar ticle-level Generation Tr u t h Pull W eights Push W eights Step 1: Reweight Sim. to Data Step 2: Reweight Gen. ν n − 1 ω n −−→ ν n AAACJ3icdVDdShtBGP1W+6Npral615vBUCgFl10bkngX6I2XFhoVsiHMTr5NBudnmZltG5Z9B1/DF/DWvoF3opfe+BydbFqo0h4YOJxzPr75TpoLbl0U3QUrq8+ev3i5tt549XrjzWbz7dax1YVhOGBaaHOaUouCKxw47gSe5gapTAWepGefF/7JNzSWa/XVzXMcSTpVPOOMOi+Nmx8TVYxLtRdXJPlh+HTmqDH6e5loiVPqncobdaQaN1tRuN/51I07JArb7YO40/Wk24kO2j0Sh1GNVn8HahyNmw/JRLNConJMUGuHcZS7UUmN40xg1UgKizllZ3SKQ08VlWhHZX1TRd57ZUIybfxTjtTq3xMlldbOZeqTkrqZfeotxH95w8JlvVHJVV44VGy5KCsEcZosCiITbpA5MfeEMsP9XwmbUUOZ8zU+2pLhXMm8avhi/lxP/k+O98M4CuMv7Va/t2wI1uAd7MIHiKELfTiEIxgAg3O4hCv4GVwE18FNcLuMrgS/Z7bhEYL7X/gNqFY= AAACJ3icdVDdShtBGP1W+6Npral615vBUCgFl10bkngX6I2XFhoVsiHMTr5NBudnmZltG5Z9B1/DF/DWvoF3opfe+BydbFqo0h4YOJxzPr75TpoLbl0U3QUrq8+ev3i5tt549XrjzWbz7dax1YVhOGBaaHOaUouCKxw47gSe5gapTAWepGefF/7JNzSWa/XVzXMcSTpVPOOMOi+Nmx8TVYxLtRdXJPlh+HTmqDH6e5loiVPqncobdaQaN1tRuN/51I07JArb7YO40/Wk24kO2j0Sh1GNVn8HahyNmw/JRLNConJMUGuHcZS7UUmN40xg1UgKizllZ3SKQ08VlWhHZX1TRd57ZUIybfxTjtTq3xMlldbOZeqTkrqZfeotxH95w8JlvVHJVV44VGy5KCsEcZosCiITbpA5MfeEMsP9XwmbUUOZ8zU+2pLhXMm8avhi/lxP/k+O98M4CuMv7Va/t2wI1uAd7MIHiKELfTiEIxgAg3O4hCv4GVwE18FNcLuMrgS/Z7bhEYL7X/gNqFY= AAACJ3icdVDdShtBGP1W+6Npral615vBUCgFl10bkngX6I2XFhoVsiHMTr5NBudnmZltG5Z9B1/DF/DWvoF3opfe+BydbFqo0h4YOJxzPr75TpoLbl0U3QUrq8+ev3i5tt549XrjzWbz7dax1YVhOGBaaHOaUouCKxw47gSe5gapTAWepGefF/7JNzSWa/XVzXMcSTpVPOOMOi+Nmx8TVYxLtRdXJPlh+HTmqDH6e5loiVPqncobdaQaN1tRuN/51I07JArb7YO40/Wk24kO2j0Sh1GNVn8HahyNmw/JRLNConJMUGuHcZS7UUmN40xg1UgKizllZ3SKQ08VlWhHZX1TRd57ZUIybfxTjtTq3xMlldbOZeqTkrqZfeotxH95w8JlvVHJVV44VGy5KCsEcZosCiITbpA5MfeEMsP9XwmbUUOZ8zU+2pLhXMm8avhi/lxP/k+O98M4CuMv7Va/t2wI1uAd7MIHiKELfTiEIxgAg3O4hCv4GVwE18FNcLuMrgS/Z7bhEYL7X/gNqFY= AAACJ3icdVDdShtBGJ21tmpsa6yX3gwGQYQuuxqSeCd446UF8wPZEGYn3yaD87PMzKrLsu/Q1+gL9La+gXeil73xOZxsItTSHhg4nHM+vvlOnHJmbBA8eSvvVt9/WFvfqG1+/PR5q779pWdUpil0qeJKD2JigDMJXcssh0GqgYiYQz++Opv7/WvQhil5afMURoJMJUsYJdZJ4/phJLNxIb+GJY5uNZvOLNFa3RSREjAlzimdUUXKcb0R+Eet43bYwoHfbJ6ErbYj7VZw0uzg0A8qNNASF+P6czRRNBMgLeXEmGEYpHZUEG0Z5VDWosxASugVmcLQUUkEmFFR3VTifadMcKK0e9LiSv1zoiDCmFzELimInZm/vbn4L2+Y2aQzKphMMwuSLhYlGcdW4XlBeMI0UMtzRwjVzP0V0xnRhFpX45stCeRSpGXNFfN6Pf4/6R35YeCH35qN086yonW0i/bQAQpRG52ic3SBuoii7+gn+oXuvB/evffgPS6iK95yZge9gff7BX9hqAI= ν n − 1 Data −− −→ ω n AAACenicdVFLa9wwEJbdV7J9ZJP01ovIUmhpauTa+7ottIceU+gmgbVZZK3sFdHDSHLaxeiH9lzor+ihWm8KTdoOCD6+mW++mVFRc2YsQt+C8N79Bw8f7e33Hj95+uygf3h0blSjCZ0TxZW+LLChnEk6t8xyellrikXB6UVx9X6bv7im2jAlP9tNTXOBK8lKRrD11LLv2qxrstBVkbcoGo+mcTo+RVEyTJJ06AEaTZIUuUw2y1a+jR3MvmpWrS3WWn1pMy3gnRaoi9O/gPuALXZerwStsG/m3LI/QNFolKTJBHpLNEZpDLeWw+k0hfGNcDB7Dro4W/Z/ZCtFGkGlJRwbs4hRbfMWa8sIp66XNYbWmFzhii48lFhQk7fdeA6+9MwKlkr7Jy3s2D8VLRbGbEThKwW2a3M3tyX/lVs0tpzkLZN1Y6kkO6Oy4dAquL05XDFNieUbDzDRzM8KyRprTKz/mVsuJd1IUbueP8zv7eH/wfm7KEZR/CkdzCa7C4E98AKcgFcgBmMwAx/BGZgDAr4H+8FRcBz8DE/C1+GbXWkY3GiOwa0I0181Z7sO AAACenicdVFLa9wwEJbdV7J9ZJP01ovIUmhpauTa+7ottIceU+gmgbVZZK3sFdHDSHLaxeiH9lzor+ihWm8KTdoOCD6+mW++mVFRc2YsQt+C8N79Bw8f7e33Hj95+uygf3h0blSjCZ0TxZW+LLChnEk6t8xyellrikXB6UVx9X6bv7im2jAlP9tNTXOBK8lKRrD11LLv2qxrstBVkbcoGo+mcTo+RVEyTJJ06AEaTZIUuUw2y1a+jR3MvmpWrS3WWn1pMy3gnRaoi9O/gPuALXZerwStsG/m3LI/QNFolKTJBHpLNEZpDLeWw+k0hfGNcDB7Dro4W/Z/ZCtFGkGlJRwbs4hRbfMWa8sIp66XNYbWmFzhii48lFhQk7fdeA6+9MwKlkr7Jy3s2D8VLRbGbEThKwW2a3M3tyX/lVs0tpzkLZN1Y6kkO6Oy4dAquL05XDFNieUbDzDRzM8KyRprTKz/mVsuJd1IUbueP8zv7eH/wfm7KEZR/CkdzCa7C4E98AKcgFcgBmMwAx/BGZgDAr4H+8FRcBz8DE/C1+GbXWkY3GiOwa0I0181Z7sO AAACenicdVFLa9wwEJbdV7J9ZJP01ovIUmhpauTa+7ottIceU+gmgbVZZK3sFdHDSHLaxeiH9lzor+ihWm8KTdoOCD6+mW++mVFRc2YsQt+C8N79Bw8f7e33Hj95+uygf3h0blSjCZ0TxZW+LLChnEk6t8xyellrikXB6UVx9X6bv7im2jAlP9tNTXOBK8lKRrD11LLv2qxrstBVkbcoGo+mcTo+RVEyTJJ06AEaTZIUuUw2y1a+jR3MvmpWrS3WWn1pMy3gnRaoi9O/gPuALXZerwStsG/m3LI/QNFolKTJBHpLNEZpDLeWw+k0hfGNcDB7Dro4W/Z/ZCtFGkGlJRwbs4hRbfMWa8sIp66XNYbWmFzhii48lFhQk7fdeA6+9MwKlkr7Jy3s2D8VLRbGbEThKwW2a3M3tyX/lVs0tpzkLZN1Y6kkO6Oy4dAquL05XDFNieUbDzDRzM8KyRprTKz/mVsuJd1IUbueP8zv7eH/wfm7KEZR/CkdzCa7C4E98AKcgFcgBmMwAx/BGZgDAr4H+8FRcBz8DE/C1+GbXWkY3GiOwa0I0181Z7sO AAACenicdVFNa9wwEJXdr2T7tU2OvYgshZamRq69X7dAe+gxhW4SWJtlrJW9IpJsJDnpYvRDey70V/RQ7WYLTdoOCB5v5s2bGRWN4MYS8i0I791/8PDR3n7v8ZOnz573XxycmbrVlM1oLWp9UYBhgis2s9wKdtFoBrIQ7Ly4/LDJn18xbXitvth1w3IJleIlp2A9tei7Lts2meuqyDsSjUfTOB0fkygZJkk69ICMJklKXKbaRafexQ5nXzWvVha0rq+7TEt8pwXZxvFfwH0EC87ra8kq8M2cW/QHJBqNkjSZYG9JxiSN8cZyOJ2mON4JB2gXp4v+j2xZ01YyZakAY+YxaWzegbacCuZ6WWtYA/QSKjb3UIFkJu+24zn8yjNLXNbaP2Xxlv1T0YE0Zi0LXynBrszd3Ib8V27e2nKSd1w1rWWK3hiVrcC2xpub4yXXjFqx9gCo5n5WTFeggVr/M7dcSrZWsnE9f5jf2+P/g7P3UUyi+HM6OJnsTrSHXqIj9BrFaIxO0Cd0imaIou/BfnAQHAY/w6PwTfj2pjQMdppDdCvC9Be8rLq6 FIG. 1. An illustration of OmniFold , applied to a set of syn- thetic and natural data. As a first step, starting from prior w eights ν 0 , the detector-level syn thetic data (“simulation”) is rew eighted to matc h the detector-level natural data (simply “data”). These weigh ts ω 1 are pulled back to induce w eights on the particle-level synthetic data (“generation”). As a sec- ond step, the initial generation is reweigh ted to matc h the new w eighted generation. The resulting w eigh ts ν 1 are pushed for- w ard to induce a new simulation, and the pro cess is iterated. induced truth is: ν 1 ( t ) p Gen. ( t ) = Z dm 0 p Gen. | Sim. ( t | m 0 ) p Data ( m 0 ) . (4) This is a con tin uous version of IBU from Eq. ( 2 ), where the sum has b een promoted to a full phase-space inte- gral. In fact, OmniFold (and IBU) are iterativ e strate- gies that conv erge to the maxim um likelihoo d estimate of the true particle-level distribution [ 27 – 31 ], whic h we discuss in detail in the App endix. After n iterations, the unfolded distribution is: p ( n ) unfolded ( t ) = ν n ( t ) p Gen. ( t ) . (5) The unfolded result can b e presented either as a set of generated even ts { t } with w eights { ν n ( t ) } (and uncer- tain ties) or, more compactly , as the learned weigh ting function ν n and instructions for sampling from p Gen. . T o demonstrate the versatilit y and p ow er of Omni- F old , we p erform a pro of-of-concept study relev ant for the LHC. Sp ecifically , we unfold the full radiation pat- tern (i.e. full phase space) of jets, which are collimated spra ys of particles arising from the fragmentation and hadronization of high-energy quarks and gluons. Jets are an ideal environmen t in whic h to b enc hmark unfold- ing techniques, since detector effects often account for a significan t p ortion of the exp erimental measurement uncertain ties for many jet substructure observ ables [ 32 ]. With the radiation pattern unfolded, one can obtain the 3 0 . 00 0 . 01 0 . 02 0 . 03 0 . 04 0 . 05 0 . 06 Normalized Cross Section [GeV − 1 ] D/T : Her wig 7.1.5 default S/G : Pythia 8.243 tune 26 Delphes 3.4.2 CMS Detector Z +jet: p Z T > 200 GeV, R = 0 . 4 “Data” Sim. IBU m “T ruth” Gen. OmniF old 0 20 40 60 Jet Mass m [GeV] 0.85 1.0 1.15 Ratio to T ruth 0 . 00 0 . 01 0 . 02 0 . 03 0 . 04 0 . 05 0 . 06 Normalized Cross Section D/T : Her wig 7.1.5 default S/G : Pythia 8.243 tune 26 Delphes 3.4.2 CMS Detector Z +jet: p Z T > 200 GeV, R = 0 . 4 “Data” Sim. IBU M “T ruth” Gen. OmniF old 0 20 40 60 80 Jet Constituen t Multiplicit y M 0.85 1.0 1.15 Ratio to T ruth 0 2 4 6 8 10 Normalized Cross Section D/T : Her wig 7.1.5 default S/G : Pythia 8.243 tune 26 Delphes 3.4.2 CMS Detector Z +jet: p Z T > 200 GeV, R = 0 . 4 “Data” Sim. IBU w “T ruth” Gen. OmniF old 0 . 0 0 . 1 0 . 2 0 . 3 0 . 4 0 . 5 0 . 6 Jet Width w 0.85 1.0 1.15 Ratio to T ruth 0 . 00 0 . 05 0 . 10 0 . 15 0 . 20 0 . 25 0 . 30 Normalized Cross Section D/T : Her wig 7.1.5 default S/G : Pythia 8.243 tune 26 Delphes 3.4.2 CMS Detector Z +jet: p Z T > 200 GeV, R = 0 . 4 “Data” “T ruth” Sim. Gen. IBU ln ρ OmniF old − 14 − 12 − 10 − 8 − 6 − 4 − 2 Soft Drop Jet Mass ln ρ 0.85 1.0 1.15 Ratio to T ruth 0 . 0 0 . 5 1 . 0 1 . 5 2 . 0 2 . 5 3 . 0 Normalized Cross Section D/T : Her wig 7.1.5 default S/G : Pythia 8.243 tune 26 Delphes 3.4.2 CMS Detector Z +jet: p Z T > 200 GeV, R = 0 . 4 “Data” “T ruth” Sim. Gen. IBU τ ( β =1) 21 OmniF old 0 . 0 0 . 2 0 . 4 0 . 6 0 . 8 1 . 0 1 . 2 N -sub jettiness Ratio τ ( β =1) 21 0.85 1.0 1.15 Ratio to T ruth 0 2 4 6 8 Normalized Cross Section D/T : Her wig 7.1.5 default S/G : Pythia 8.243 tune 26 Delphes 3.4.2 CMS Detector Z +jet: p Z T > 200 GeV, R = 0 . 4 “Data” Sim. IBU z g “T ruth” Gen. OmniF old 0 . 0 0 . 1 0 . 2 0 . 3 0 . 4 0 . 5 Gro omed Jet Momen tum F raction z g 0.85 1.0 1.15 Ratio to T ruth FIG. 2. The unfolding results for six jet substructure observ ables, using Her wig 7.1.5 (“Data”/“T ruth”) and Pythia 8.243 tune 26 (Sim./Gen.), unfolded with OmniFold and compared to IBU. OmniFold matches or exceeds the unfolding p erformance of IBU on all of these observ ables. W e emphasize that OmniFold is a single general unfolding pro cedure, whereas unfolding with IBU must b e done observ able by observ able. Statistical uncertainties are shown only in the ratio panel. unfolded distribution of any observ able using Eq. ( 5 ). Hence, this procedure can be view ed as simultaneously unfolding all observ ables. Our study is based on proton-proton collisions gener- ated at √ s = 14 T eV with the default tune of Her- wig 7.1.5 [ 33 – 35 ] and T une 26 [ 36 ] of Pythia 8.243 [ 37 – 39 ] in order to study a c hallenging setting where the “nat- ural” and “syn thetic” distributions are substantially dif- feren t. As a proxy for detector effects and a full detector sim ulation, we use the Delphes 3.4.2 [ 40 ] fast simula- tion of the CMS detector, whic h uses particle flo w re- construction. Jets with radius parameter R = 0 . 4 are clustered using either all particle flow ob jects (detector- lev el) or stable non-neutrino truth particles (particle- lev el) with the anti- k T algorithm [ 41 ] implemented in F astJet 3.3.2 [ 42 , 43 ]. One of the sim ulations ( Her- wig ) plays the role of “data”/“truth”, while the other ( Pythia ) is used to derive the unfolding corrections. T o reduce acceptance effects, the leading jets are studied in even ts with a Z b oson with transverse momentum p Z T > 200 GeV. After applying the selections, we obtain appro ximately 1.6 million even ts from eac h generator. An y suitable mac hine learning arc hitecture can be used for OmniFold . F or this study , we use Particle Flow Net works (PFNs) [ 44 , 45 ] to process jets in their natu- ral represen tation as sets of particles. Intuitiv ely , PFNs learn and pro cesses a set of additiv e observ ables via PFN( { p i } M i =1 ) = F  P M i =1 Φ( p i )  for an ev en t with M particles p i , where F and Φ are parameterized by fully- connected netw orks. W e sp ecify the particles by their transv erse momentum p T , rapidity y , azimuthal angle φ , and particle identification co de [ 46 ], restricted to the exp erimen tally-accessible information (PFN-Ex [ 44 ]) at detector-lev el. T o define separate mo dels for Step 1 and Step 2 , we use the PFN arc hitecture and training param- eters of Ref. [ 44 ] with latent space dimension ` = 256, implemen ted in the Ener gyFlow Python pack age [ 47 ]. Neural netw orks are trained with Keras [ 48 ] and T ensor- Flo w [ 49 ] using the Adam [ 50 ] optimization algorithm. The mo dels are randomly initialized in the first iteration and subsequen tly w arm-started using the model from the previous iteration. 20% of the ev en ts are reserved as a v alidation set during training. T o inv estigate the unfolding p erformance, w e consider six widely-used jet substructure observ ables [ 51 ]. The first four are jet mass m , constituen t m ultiplicity M , the 4 Observ able Metho d m M w ln ρ τ 21 z g OmniF old 2.77 0.33 0.10 0.35 0.53 0.68 Mul tiFold 3.80 0.89 0.09 0.37 0.26 0.15 UniF old 8.82 1.46 0.15 0.59 1.11 0.59 IBU 9.31 1.51 0.11 0.71 1.10 0.37 Data 24.6 130 15.7 14.2 11.1 3.76 Generation 3.62 15 22.4 19 20.8 3.84 T ABLE I. The unfolding p erformance of OmniF old , Mul ti- F old , and UniF old on six jet substructure observ ables, com- pared to IBU. The performance is quantified b y the triangular discriminator [ 60 – 62 ] ∆( p, q ) = 1 2 R dλ ( p ( λ ) − q ( λ )) 2 p ( λ )+ q ( λ ) ( × 10 3 ) b e- t ween the unfolded and truth-level (binned) histograms. Also sho wn are the distances from data (no unfolding) and gen- eration (the prior). The b est unfolding metho d for each ob- serv able is shown in bold. All metho ds p erform well, with OmniF old providing consistently go o d performance. N -sub jettiness ratio τ 21 = τ ( β =1) 2 /τ ( β =1) 1 [ 52 , 53 ], and the jet width w (implemen ted as τ ( β =1) 1 ). Since jet gro om- ing [ 54 – 58 ] is of recen t in terest, we also sho w the jet mass ln ρ = ln m 2 SD /p 2 T and momentum fraction z g after Soft Drop gro oming [ 57 , 58 ] with z cut = 0 . 1 and β = 0. Sev- eral of these observ ables are computed with the help of F astJet Contrib 1.042 [ 59 ]. The unfolding p erformance of OmniFold is sho wn in Fig. 2 and compared to IBU, b oth with n = 5 iterations. W e found little difference b etw een n = 3 and n = 5, though OmniFold exhibits a slight preference for more iterations. OmniF old succeeds in simultaneously un- folding all of these observ ables, achieving p erformance comparable to or better than IBU applied to eac h observ- able individually . The mass is challenging for all meth- o ds as particle-type information is relev ant at particle- lev el but is not fully known at detector-lev el, in tro ducing additional prior dep endence. Though OmniFold is un- binned, the data are only able to constrain energy and angular scales comparable to the detector resolution. Statistical uncertain ties from the prior distribution are sho wn in the b ottom panels of Fig. 2 , holding the un- folding procedure (i.e. resp onse matrix and rew eighting) fixed. F or this proof-of-concept study , we do not show systematic uncertain ties, though the pro cedure for deriv- ing them is the same as for IBU. Non-closure and mo del- ing uncertainties can b e derived in the standard w ay by testing the pro cedure on differen t Monte Carlo samples and comparing the results to the kno wn “truth” distri- butions. (W e chec k ed that OmniFold satisfies technical closure when Pythia is unfolded to itself.) Exp erimen- tal systematic uncertainties can be obtained b y v arying the relev an t effects and rep eating the unfolding proce- dure. Like other unfolding pro cedures, OmniFold can- not impro ve the results in phase-space regions that are unconstrained b y observed quan tities. It can, how ev er, impro ve the p erformance if the full phase space contains 0 1 2 3 4 5 6 7 Correlation Dimension D/T : Her wig 7.1.5 default S/G : Pythia 8.243 tune 26 Delphes 3.4.2 CMS Detector Z +jet: p Z T > 200 GeV, R = 0 . 4 p jet T > 500 GeV, scaled to 500 GeV “Data” Sim. OmniF old “T ruth” Gen. 10 1 10 2 Energy Scale Q [GeV] 0.8 1.0 1.2 Ratio to T ruth FIG. 3. The correlation dimension of the space of jets, un- folded with OmniFold . The unfolded results closely match the truth-lev el dimension o v er most of the energy range, tend- ing tow ard the prior in the more difficult phase space region at lo w Q . Unfolding a complicated statistic suc h as the cor- relation dimension is challenging with standard metho ds. auxiliary features relev ant for the detector resp onse. T o capitalize on this full phase-space approac h, it is essen- tial that the detector simulation properly describes these features and that systematic uncertain ties are estimated using a high-dimensional approach [ 63 , 64 ]. T o highligh t the flexibilit y of our unfolding framew ork, w e study v ariations of OmniFold , where the av ailable information is v aried b y con trolling the inputs: • UniFold : A single observ able as input. This is an un binned version of IBU. • Mul tiF old : Many observ ables as input. Here, w e use the six jet substructure observ ables in Fig. 2 to deriv e the detector resp onse. • OmniFold : The full ev ent (or jet) as input, using the full phase space information. The unfolding p erformance of each metho d on our six substructure observ ables is tabulated in T able I and com- pared to IBU. The UniFold and Mul tiF old implemen- tations b oth use dense netw orks with three lay ers of one h undred no des each and a t wo-node output la y er. W e see go o d unfolding p erformance across all metho ds, and ev en though OmniF old is not directly trained on these six observ ables, it p erforms comparably to or b etter than Mul tiFold . While the detector resp onse dep ends on the jet rapidit y , we chec k ed that Mul tiFold did not significan tly b enefit from including the rapidity , though doing so could be important in a real experimental con- text. In general, additional information can be included 5 and the unfolding pro cedure can be repeated, with the final model chosen as the one with the best detector-level agreemen t with the data. Since OmniFold unfolds the full radiation pattern, it can b e used to prob e new, physically-in teresting quan ti- ties that are challenging to unfold with existing metho ds. One example is the recently-proposed correlation (frac- tal) dimension of jets [ 65 , 66 ], whic h is a function of the energy scale Q . This complicated statistic is defined by pairwise metric distances b et w een jet radiation patterns, falling outside of the purview of single-observ able unfold- ing tec hniques. Within our jet samples, we restrict to energetic jets with p jet T > 500 GeV, b o osted to the ori- gin of the rapidity-azim uth plane, and with constituents rescaled to ha ve p T summing to 500 GeV. The correlation dimensions of these jets, b oth b efore and after applying OmniF old , are shown in Fig. 3 . The unfolded results matc h the true distribution ov er a wide range of Q v al- ues, with residual prior dep endence seen in at low Q (i.e. the infrared) where jets hav e a higher dimensionality and detector effects ha ve a larger impact, thus making the unfolding problem more difficult. More broadly , Omni- F old op ens the do or to going b eyond p er-ev ent collider observ ables to w ards more n uanced or intricate measure- men ts of the data. In conclusion, w e ha v e presen ted a p oten tially transfor- mativ e unfolding paradigm based on iteratively rew eight- ing a set of simulated even ts with mac hine learning. Our OmniF old approach allows an en tire dataset to be un- folded using all of the av ailable information, av oiding the need for binning and restricting to single observ ables. W e ha ve demonstrated the pow er of this metho d in a (sim- ulated) case of interest by unfolding the full radiation pattern of jets, paving the w a y for significant adv ances in jet substructure at the LHC. Our unfolding frame- w ork allow ed us to go b ey ond p er-ev ent observ ables to- w ards unfolding more complex dataset statistics, suc h as fractal dimensions of the space of jets. Going even further, (unsup ervised) machine learning mo dels may b e trained directly at particle-level b y using the unfolded and weigh ted dataset, which is a fascinating a ven ue for further exploration. These adv ances ha ve broad applica- bilit y b eyond particle ph ysics in domains where deconv o- lution or unfolding is used, suc h as image-based measure- men ts and quan tum computation [ 67 ]. T o enable future unfolding studies and developmen ts, we hav e made our co de and jet datasets publicly a v ailable [ 68 , 69 ], including t wo additional tunes of Pythia b ey ond those presen ted here. Finally , our reweigh ting-based unfolding strategy allo ws for new observ ables to b e measured long after the unfolding is carried out, which can significan tly emp o wer future public and archiv al collider data analyses [ 70 ]. W e are grateful to Gabriel Collin, Christian Herwig, Andrew Larkoski, Matthew LeBlanc, Salv atore Rapp o c- cio, Jennifer Roloff, Matthew Sch w artz, Daniel White- son, and Mike Williams for helpful comments and dis- cussions. W e also thank the referees for their v alu- able suggestions, including the clarification of the sta- tistical framing of the metho d. The authors b enefited from the hospitality of the Harv ard Center for the F un- damen tal Laws of Nature. This work was partially completed at the Asp en Center for Physics, whic h is supp orted b y National Science F oundation grant PHY- 1607611. AA and BN w ere supp orted by the U.S. De- partmen t of Energy (DOE), Office of Science under con- tract DE-AC02-05CH11231. PTK, EMM, and JT w ere supp orted by the DOE Office of Nuclear Physics under Gran t No. DE-SC0011090 and the DOE Office of High Energy Ph ysics under Grant Nos. DE-SC0012567 and DE-SC0019128. JT is additionally supp orted by the Si- mons F oundation through a Simons F ellowship in The- oretical Physics. Cloud computing resources were pro- vided through a Google Cloud allotment from the MIT Quest for Intelligence. BN w ould like to thank NVIDIA for pro viding V olta GPUs for neural net work training. Emily Dickinson, #975 The Moun tain sat up on the Plain In his tremendous Chair – His observ ation omnifold, His inquest, everywhere – The Seasons play ed around his knees Lik e Children round a sire – Grandfather of the Days is He Of Da wn, the Ancestor – App endix: OmniFold as a Maxim um Likelihoo d Estimate In this App endix, we review the statistical underpinnings of Iterativ e Bay esian Unfolding (IBU) [ 5 ] as w ell as OmniF old and confirm that they conv erge to the maximum lik eliho od estimate of the true particle-lev el distribution. This discussion serv es to clarify the statistical formulation of unfolding, as w ell as to provide a deriv ation of the correctness of OmniF old . W e follow the ov erall spirit of Ref. [ 27 ], k eeping the form ulation general and un binned, w orking in the asymptotic limit of large amounts of data. W e seek to find the truth-level reweigh ting ν ( t ) of the syn thetic particle-level distribution p Gen. ( t ) that maximizes the likelihoo d of observing the measured data. 6 The (log) likelihoo d of a given reweigh ting to pro duce the observ ed data is: LL[ ν ] = Z dm p Data ( m ) ln  Z dt 0 p ( m | t 0 ) ν ( t 0 ) p Gen. ( t 0 )  − λ  Z dt 0 ν ( t 0 ) p Gen. ( t 0 ) − 1  . (6) Here, p Data ( m ) is the measured distribution of data at detector lev el. The quantit y p ( m | t ) captures the detector resp onse, i.e. the distribution of the detector-lev el information m produced b y the particle-lev el information t . W e tak e the detector resp onse to b e accurately mo deled in the synthetic dataset, p ( m | t ) = p Data | T ruth ( m | t ) = p Sim. | Gen. ( m | t ), one of the standard assumptions of unfolding. The last term is a Lagrange m ultiplier constrain t to ensure that the particle-lev el reweigh ting ν ( t ) of the synthetic distribution p Gen. ( t ) yields a normalized distribution. T o maximize the likelihoo d, we v ary it with resp ect to the reweigh ting function ν : δ LL δ ν ( t ) = Z dm p Data ( m ) p ( m | t ) p Gen. ( t ) R dt 0 p ( m | t 0 ) ν ( t 0 ) p Gen. ( t 0 ) − λ p Gen. ( t ) = 0 . (7) In tegrating this equation equation against R dt ν ( t ) and applying the normalization condition yields that λ = 1. The stationary condition in Eq. ( 7 ) results in a maximum of the likelihoo d b ecause the second v ariation is non-p ositiv e and therefore the functional is conca ve: δ 2 LL δ ν ( t 0 ) δ ν ( t 1 ) = − Z dm p Data ( m ) p ( m | t 0 ) p Gen. ( t 0 ) p ( m | t 1 ) p Gen. ( t 1 )  R dt 0 p ( m | t 0 ) ν ( t 0 ) p Gen. ( t 0 )  2 ≤ 0 . (8) T o connect the maxim um lik eliho od strategy to the OmniFold and IBU metho ds, w e can multiply the stationary condition in Eq. ( 7 ) on b oth sides by ν ( t ) to obtain an equation satisfied by the optimal reweigh ting function ν ∗ ( t ): ν ∗ ( t ) p Gen. ( t ) = Z dm p Data ( m ) p ( m | t ) ν ∗ ( t ) p Gen. ( t ) R dt 0 p ( m | t 0 ) ν ∗ ( t 0 ) p Gen. ( t 0 ) . (9) If w e w ere to replace ν ∗ ( t ) on the left-hand side of Eq. ( 9 ) b y ν n ( t ) and on the right-hand side by ν n − 1 ( t ), w e w ould obtain the up date rule for OmniFold , with the discrete v ersion corresp onding to IBU: ν n ( t ) p Gen. ( t ) = Z dm p Data ( m ) p ( m | t ) ν n − 1 ( t ) p Gen. ( t ) R dt 0 p ( m | t 0 ) ν n − 1 ( t 0 ) p Gen. ( t 0 ) . (10) T o see that Eq. ( 10 ) indeed causes the likelihoo d to increase, we will show that it is a consequence of a generalized exp ectation-maximization (EM) algorithm [ 28 – 31 ] in which the likelihoo d increases at each step. W e use the particle- lev el information t as the unobserved laten t v ariables. By Theorem 1 of Ref. [ 28 ], w e only need to show that the exp ected complete-data (log) lik eliho o d Q increases from one choice of reweigh ting ν n − 1 ( t ) to another ν n ( t ): Q ( ν n | ν n − 1 ) = Z dm p Data ( m ) Z dt p ( t | m, ν n − 1 ) ln p ( t, m | ν n ) . (11) Manipulating the argument of Eq. ( 11 ) using conditional probabilities and Bay es’ rule, we hav e: Q ( ν n | ν n − 1 ) = Z dm p Data ( m ) Z dt p ( m | t ) ν n − 1 ( t ) p Gen. ( t ) R dt 0 p ( m | t 0 ) ν n − 1 ( t 0 ) p Gen. ( t 0 ) ln  ν n ( t ) p Gen. ( t )  + const. , (12) where the resp onse p ( m | t ) is indep enden t of ν and p ( t | ν ) = ν ( t ) p Gen. ( t ). The constant term is indep endent of ν n and so will not contribute to our maximization ov er ν n . W e can maximize Eq. ( 12 ) with respect to ν n ( t ), including the Lagrange multiplier constrain t to enforce normal- ization, b y taking the deriv ativ e and setting it equal to zero. This collapses the integral o ver t and leads exactly to Eq. ( 10 ). Thus the c hoice of ν n ( t ) via Eq. ( 10 ) increases Q , and so the log lik eliho od also increases by the prop erties of the generalized EM algorithm. While the maxim um may not be unique due to null directions in Eq. ( 8 ), a global maxim um of the likelihoo d is attained due to its concavit y , with the precise ν ∞ ( t ) dictated b y the c hoice of initial distribution p Gen. ( t ). If the resp onse “matrix” p ( m | t ) is inv ertible, this pro cedure conv erges to the true solution. F ur- ther, terminating the algorithm after a finite n umber of iterations introduces regularization b y reducing the v ariance of the estimator at the cost of increased bias via prior dependence. Th us, OmniFold pro vides an unbinned, machine-learning-based strategy to estimate the maxim um lik eliho o d true particle-lev el distribution giv en the observed detector-lev el data. Of course, in the presence of statistical fluctuations, the maxim um of the likelihoo d ma y not b e the desired solution, so in practice, one regularizes the unfolding pro cedure b y p erforming a finite num ber of iterations. 7 ∗ a jandreassen@go ogle.com † pk omiske@mit.edu ‡ meto diev@mit.edu § bpnac hman@lbl.gov ¶ jthaler@mit.edu [1] G. Cow an, “A surv ey of unfolding metho ds for particle ph ysics,” Conf. Pro c. C0203181 , 248 (2002). [2] V olker Blob el, “Unfolding Methods in Particle Physics,” PHYST A T2011 Pro ceedings , 240 (2011) . [3] V olker Blob el, “Unfolding,” Data Analysis in High En- ergy Ph ysics , 187 (2013) . [4] Rahul Balasubramanian, Lydia Brenner, Carsten Bur- gard, Glen Co wan, Vincen t Croft, W outer V erk erke, and Pim V ersc huuren, “Statistical metho d and comparison of different unfolding techniques using Ro oFit,” (2019), arXiv:1910.14654 [ph ysics.data-an] . [5] G. D’Agostini, “A Multidimensional unfolding method based on Ba yes’ theorem,” Nucl. Instrum. Meth. A362 , 487–498 (1995) . [6] Andreas Ho c ker and V akh tang Kartv elishvili, “SVD ap- proac h to data unfolding,” Nucl. Instrum. Meth. A372 , 469–481 (1996) , arXiv:hep-ph/9509307 [hep-ph] . [7] Stefan Sc hmitt, “TUnfold: an algorithm for correct- ing migration effects in high energy physics,” JINST 7 , T10003 (2012) , arXiv:1205.6201 [physics.data-an] . [8] Nikolai D. Gagunashvili, “Machine learning approach to in verse problem and unfolding pro cedure,” (2010), arXiv:1004.2006 [ph ysics.data-an] . [9] Alexander Glazov, “Machine learning as an instru- men t for data unfolding,” (2017), [ph ysics.data-an] . [10] Kaustuv Datta, Deepak Kar, and Debarati Roy , “Un- folding with Generativ e Adversarial Netw orks,” (2018), arXiv:1806.00433 [ph ysics.data-an] . [11] G. Zec h and B. Aslan, “Binning-F ree Unfolding Based on Mon te Carlo Migration,” Statistic al pr oblems in p ar- ticle physics, astrophysics and cosmolo gy. Pr o c e e dings, Confer enc e, PHYST A T 2003, Stanfor d, USA, Septem- b er 8-11, 2003 , eConf C030908 , TUGT001 (2003), [,138(2003)]. [12] L. Lindemann and Gun ter Zec h, “Unfolding by w eigh ting Mon te Carlo even ts,” Nucl. Instrum. Meth. A354 , 516– 521 (1995) . [13] D Martschei, M F eindt, S Honc, and J W agner-Kuhr, “Adv anced ev ent reweigh ting using multiv ariate analy- sis,” Journal of Ph ysics: Conference Series 368 , 012028 (2012) . [14] A. Rogozhniko v, “Reweigh ting with Bo osted Decision T rees,” Pr o c e e dings, 17th International Workshop on Ad- vanc e d Computing and Analysis T echniques in Physics R ese ar ch (ACA T 2016): V alp ar aiso, Chile, January 18-22, 2016 , J. Phys. Conf. Ser. 762 , 012036 (2016) , arXiv:1608.05806 [ph ysics.data-an] . [15] Ro el Aaij et al. (LHCb Collab oration), “Observ ation of the decays Λ 0 b → χ c 1 pK − and Λ 0 b → χ c 2 pK − ,” Phys. Rev. Lett. 119 , 062001 (2017) , arXiv:1704.07900 [hep- ex] . [16] M. Aaboud et al. (A TLAS Collaboration), “Searc h for pair production of higgsinos in final states with at least three b -tagged jets in √ s = 13 T eV pp collisions using the A TLAS detector,” Phys. Rev. D98 , 092002 (2018) , arXiv:1806.04030 [hep-ex] . [17] Anders Andreassen, Ily a F eige, Christopher F rye, and Matthew D. Sc hw artz, “JUNIPR: a F ramework for Un- sup ervised Machine Learning in P article Physics,” Eur. Ph ys. J. C79 , 102 (2019) , arXiv:1804.09720 [hep-ph] . [18] Anders Andreassen and Benjamin Nachman, “Neural Net works for F ull Phase-space Reweigh ting and Param- eter T uning,” (2019), arXiv:1907.08209 [hep-ph] . [19] Emily Dic kinson, “The Moun tain Sat Upon the Plain,” in The Complete Po ems of Emily Dickinson (Little, Brown, and Compan y , 1924). [20] L. B. Lucy, “An iterative tec hnique for the rectification of observed distributions,” Astronomical Journal 79 , 745 (1974) . [21] William Hadley Ric hardson, “Ba yesian-based iterative metho d of image restoration,” J. Opt. So c. Am. 62 , 55– 59 (1972) . [22] Kyle Cranmer, Juan P av ez, and Gilles Louppe, “Ap- pro ximating Likelihoo d Ratios with Calibrated Discrim- inativ e Classifiers,” (2015), arXiv:1506.02169 [stat.AP] . [23] Johann Brehmer, Kyle Cranmer, Gilles Louppe, and Juan Pa vez, “Constraining Effective Field Theories with Mac hine Learning,” Phys. Rev. Lett. 121 , 111801 (2018) , arXiv:1805.00013 [hep-ph] . [24] Johann Brehmer, Kyle Cranmer, Gilles Louppe, and Juan P av ez, “A Guide to Constraining Effectiv e Field Theories with Machine Learning,” Phys. Rev. D98 , 052004 (2018) , arXiv:1805.00020 [hep-ph] . [25] Johann Brehmer, Gilles Loupp e, Juan Pa v ez, and Kyle Cranmer, “Mining gold from implicit mo dels to improv e lik eliho o d-free inference,” (2018), [stat.ML] . [26] Enrico Bothmann and Luigi Del Debbio, “Reweigh ting a parton show er using a neural net work: the final-state case,” Journal of High Energy Physics 2019 , 33 (2019) . [27] L. A. Shepp and Y. V ardi, “Maximum likelihoo d recon- struction for emission tomography ,” IEEE transactions on medical imaging 1 , 113–122 (1982) . [28] A. P . Dempster, N. M. Laird, and D. B. Rubin, “Max- im um lik eliho od from incomplete data via the EM algo- rithm,” Journal of the Ro yal Statistical So ciety . Series B (Metho dological) 39 , 1–38 (1977) . [29] C. F. Jeff W u, “On the conv ergence prop erties of the EM algorithm,” Ann. Statist. 11 , 95–103 (1983) . [30] Y. V ardi, L. A. Shepp, and L. Kaufman, “A statistical mo del for p ositron emission tomography ,” Journal of the American Statistical Asso ciation 80 , 8–20 (1985) . [31] Mik ael Kuusela, “Statistical issues in unfolding metho ds for high energy physics,” Aalto Universit y Master’s The- sis (2012) . [32] Roman Kogler et al. , “Jet Substructure at the Large Hadron Collider: Exp erimental Review,” Rev. Mo d. Ph ys. 91 , 045003 (2019) , arXiv:1803.06991 [hep-ex] . [33] M. Bahr et al. , “Herwig++ Physics and Man ual,” Eur. Ph ys. J. C58 , 639–707 (2008) , arXiv:0803.0883 [hep-ph] . [34] Johannes Bellm et al. , “Herwig 7.0/Herwig++ 3.0 release note,” Eur. Phys. J. C76 , 196 (2016) , [hep-ph] . [35] Johannes Bellm et al. , “Herwig 7.1 Release Note,” (2017), arXiv:1705.06919 [hep-ph] . [36] A TLAS Run 1 Pythia8 tunes , T ech. Rep. A TL-PHYS- PUB-2014-021 (CERN, Genev a, 2014). [37] T orb jorn Sj¨ ostrand, Stephen Mrenna, and Peter Z. Sk ands, “A Brief In troduction to PYTHIA 8.1,” Comput. 8 Ph ys. Commun. 178 , 852–867 (2008) , [hep-ph] . [38] T orb jorn Sj¨ ostrand, Stephen Mrenna, and Peter Z. Sk ands, “PYTHIA 6.4 Physics and Manual,” JHEP 05 , 026 (2006) , arXiv:hep-ph/0603175 [hep-ph] . [39] T orb jorn Sj¨ ostrand, Stefan Ask, Jesp er R. Chris- tiansen, Richard Corke, Nishita Desai, Philip Ilten, Stephen Mrenna, Stefan Prestel, Christine O. Ras- m ussen, and Peter Z. Sk ands, “An Introduction to PYTHIA 8.2,” Comput. Phys. Commun. 191 , 159–177 (2015) , arXiv:1410.3012 [hep-ph] . [40] J. de F av ereau, C. Delaere, P . Demin, A. Giammanco, V. Lematre, A. Mertens, and M. Selv aggi (DELPHES 3), “DELPHES 3, A mo dular framew ork for fast sim ulation of a generic collider exp erimen t,” JHEP 02 , 057 (2014) , arXiv:1307.6346 [hep-ex] . [41] Matteo Cacciari, Gavin P . Salam, and Gregory Soy ez, “The Anti-k(t) jet clustering algorithm,” JHEP 04 , 063 (2008) , arXiv:0802.1189 [hep-ph] . [42] Matteo Cacciari, Gavin P . Salam, and Gregory Soy ez, “F astJet User Man ual,” Eur. Phys. J. C72 , 1896 (2012) , arXiv:1111.6097 [hep-ph] . [43] Matteo Cacciari and Gavin P . Salam, “Dispelling the N 3 m yth for the k t jet-finder,” Ph ys. Lett. B641 , 57– 61 (2006) , arXiv:hep-ph/0512210 [hep-ph] . [44] Patric k T. Komiske, Eric M. Metodiev, and Jesse Thaler, “Energy Flow Netw orks: Deep Sets for Particle Jets,” JHEP 01 , 121 (2019) , arXiv:1810.05165 [hep-ph] . [45] Manzil Zaheer, Sat wik Kottur, Siamak Rav anbakhsh, Barnab´ as P´ oczos, Ruslan Salakhutdino v, and Alexan- der J. Smola, “Deep sets,” in A dvanc es in Neur al In- formation Pr o c essing Systems 30: Annual Confer enc e on Neur al Information Pr o c essing Systems 2017, 4-9 De c em- b er 2017, L ong Be ach, CA, USA (2017) pp. 3391–3401. [46] M. T anabashi et al. (P article Data Group), “Review of P article Physics,” Phys. Rev. D98 , 030001 (2018) . [47] Patric k T. Komiske and Eric M. Metodiev, “Ener- gyFlo w,” (2019), https://energyflow.network/ . [48] F rancois Chollet, “Keras,” https://github.com/ fchollet/keras (2017). [49] Mart ´ ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Da vis, Jeffrey Dean, Matthieu Devin, Sanja y Ghema wat, Geoffrey Irving, Mic hael Isard, et al. , “T en- sorflo w: A system for large-scale machine learning.” in OSDI , V ol. 16 (2016) pp. 265–283. [50] Diederik Kingma and Jimmy Ba, “Adam: A metho d for sto c hastic optimization,” (2014), arXiv:1412.6980 [cs] . [51] Andrew J. Lark oski, Ian Moult, and Benjamin Nac hman, “Jet Substructure at the Large Hadron Collider: A Re- view of Recent Adv ances in Theory and Machine Learn- ing,” Ph ys. Rept. 841 , 1–63 (2020) , [hep-ph] . [52] Jesse Thaler and Ken V an Tilburg, “Identifying Bo osted Ob jects with N-sub jettiness,” JHEP 03 , 015 (2011) , arXiv:1011.2268 [hep-ph] . [53] Jesse Thaler and Ken V an Tilburg, “Maximizing Bo osted T op Identification by Minimizing N-sub jettiness,” JHEP 02 , 093 (2012) , arXiv:1108.2701 [hep-ph] . [54] David Krohn, Jesse Thaler, and Lian-T ao W ang, “Jet T rimming,” JHEP 02 , 084 (2010) , arXiv:0912.1342 [hep- ph] . [55] Stephen D. Ellis, Christopher K. V ermilion, and Jonathan R. W alsh, “Recombination Algorithms and Jet Substructure: Pruning as a T o ol for Heavy P article Searc hes,” Ph ys. Rev. D81 , 094023 (2010) , arXiv:0912.0033 [hep-ph] . [56] Stephen D. Ellis, Christopher K. V ermilion, and Jonathan R. W alsh, “T ec hniques for improv ed hea vy par- ticle searches with jet substructure,” Ph ys. Rev. D80 , 051501 (2009) , arXiv:0903.5081 [hep-ph] . [57] Mrinal Dasgupta, Alessandro F regoso, Simone Marzani, and Ga vin P . Salam, “T ow ards an understanding of jet substructure,” JHEP 09 , 029 (2013) , [hep-ph] . [58] Andrew J. Lark oski, Simone Marzani, Gregory Soy ez, and Jesse Thaler, “Soft Drop,” JHEP 05 , 146 (2014) , arXiv:1402.2657 [hep-ph] . [59] “F astjet con trib,” https://fastjet.hepforge.org/ contrib/ . [60] F. T opso e, “Some inequalities for information divergence and related measures of discrimination,” IEEE T ransac- tions on Information Theory 46 , 1602–1609 (2000) . [61] Philipp e Gras, Stefan Hche, Deepak Kar, Andrew Lark oski, Leif Lnn blad, Simon Pltzer, Andrzej Sidmok, P eter Sk ands, Gregory Soy ez, and Jesse Thaler, “Sys- tematics of quark/gluon tagging,” JHEP 07 , 091 (2017) , arXiv:1704.03878 [hep-ph] . [62] Samuel Brigh t-Thonney and Benjamin Nac hman, “In v es- tigating the T op ology Dep endence of Quark and Gluon Jets,” JHEP 03 , 098 (2019) , arXiv:1810.05653 [hep-ph] . [63] Benjamin Nachman and Chase Shimmin, “AI Safety for High Energy Physics,” (2019), arXiv:1910.08606 [hep- ph] . [64] Benjamin Nachman, “A guide for deploying Deep Learn- ing in LHC searches: Ho w to achiev e optimality and ac- coun t for uncertain ty ,” (2019), arXiv:1909.03081 [hep- ph] . [65] Patric k T. Komiske, Eric M. Metodiev, and Jesse Thaler, “Metric Space of Collider Even ts,” Phys. Rev. Lett. 123 , 041801 (2019) , arXiv:1902.02346 [hep-ph] . [66] Patric k T. Komiske, Radha Mastandrea, Eric M. Meto diev, Preksha Naik, and Jesse Thaler, “Exploring the Space of Jets with CMS Op en Data,” Phys. Rev. D 101 , 034009 (2020) , arXiv:1908.08542 [hep-ph] . [67] B. Nachman, M. Urbanek, W. A. de Jong, and C. Bauer, “Unfolding Quantum Computer Readout Noise,” (2019), arXiv:1910.01969 [quan t-ph] . [68] Anders Andreassen, Patric k T. Komiske, Eric M. Meto diev, Benjamin Nac hman, and Jesse Thaler, “Om- nifold,” https://github.com/ericmetodiev/OmniFold (2019). [69] Anders Andreassen, Patric k Komiske, Eric Meto diev, Benjamin Nachman, and Jesse Thaler, “Pythia/Herwig + Delphes Jet Datasets for OmniF old Unfolding,” Zen- o do (2019), 10.5281/zeno do.3548091 . [70] Xiaoli Chen, S ¨ unje Dallmeier-Tiessen, Robin Dasler, Se- bastian F eger, Pamfilos F okianos, Jose Benito Gonzalez, Harri Hirv onsalo, Dinos Kousidis, Artemis Lav asa, Salv a- tore Mele, Diego Rodriguez Rodriguez, Tibor ˇ Simk o, Tim Smith, Ana T risovic, Anna T rzcinsk a, Ioannis Tsanakt- sidis, Markus Zimmermann, Kyle Cranmer, Luk as Hein- ric h, Gordon W atts, Michael Hildreth, Lara Lloret Igle- sias, Kati Lassila-Perini, and Sebastian Neub ert, “Op en is not enough,” Nature Physics 15 , 113–119 (2019) .