Have a Tip or Trick to add to this page? Submit through this form.
Table of Contents
General Tips and Tricks
Making Custom Graphics and Diagrams
CDMS Tips and Tricks
DUNE Tips and Tricks
Computing
Identifying Memory Leaks
For information on tracking down memory leaks, take a look here.
Slurm “Send/Recv” Error
If you submit a Slurm job and get back a “Socket timed out” or “send/recv” error, your job probably got submitted successfully. The error means that the Slurm scheduler wasn’t able to communicate back to your login process about the job status. Don’t keep submitting jobs.
Wait a minute or so, and use:
squeue -u <username>General Analysis
Accessing Built in Jupyter Debugger
Jupyter notebook contains a built-in debugger that allows you to add break points in the code and quickly debug. This can be used without inside a regular jupyter notebook session and does not require any outside setup. See more information about the debugger here.
Disabling Root
To avoid new versions of ROOT launching a Web-based TBrowser, add these two lines to ~/.rootrc: Browser.Name: TRootBrowser
Making Profile Plots
A profile plot is used to display the mean value of Y and its error for each bin in X. It shows the changes in Y over X. Here is a quick example of how to make a 1D profile plot using the ROOT, root_numpy and matplotlib packages in Python. See the following example for details:
import matplotlib.pyplot as plt
import numpy as np
from ROOT import TProfile
from root_numpy import fill_profile
# Fill the numbers
profile =TProfile("profname","title", xbn, xbins) # xbn: number of x bins; xbins: an array
fill_array = np.column_stack((X_array, Y_array))
fill_profile(profname,fill_array)
# Read the numbers for a profile plot
profile_X = np.array([profname.GetBinCenter(xbin) for xbin in range(1,xbn+1)])
profile_mean = np.array([profname.GetBinContent(xbin) for xbin in range(1,xbn+1)])
profile_error = np.array([profname.GetBinError(xbin) for xbin in range(1,xbn+1)])
bin_entr = np.array([profname.GetBinEntries(xbin) for xbin in range(1,xbn+1)])
profile_std = np.sqrt(bin_entr)*profile_error # std of mean -> std of the data
# Plotting
plt.errorbar(profile_X, profile_mean, yerr=profile_std)
Reducing memory usage with ROOT and Numpy
When you are reading from a massive ROOT Tree, make sure you only read the specific trees and branches you need, and not all the trees or all the branches in a tree, which can lead to a memory overload
Create ROOT Kernel for your Jupyter OnDemand
“If you like using ROOT directly, you can create a ROOT kernel and run that in your Jupyter session instead of the normal JupyterLab Python kernel. You can use the basic ROOT C++ interface in a Jupyter “”In[]/Out[]”” style.
You can also put “”%%cpp”” in a regular Jupyter cell (after importing ROOT), and use C++ in that cell.”
How to make a “heat map”
“A heat map is a three-dimensional plot (or 2D histogram, or 2D profle), where the ‘Z’ axis is represent by a color range for each point or each bin.
To make a heat-map style scatter plot (assuming you use our conventional imports of matplotlib)
`plt.scatter(x,y,heat,cmap=’coolwarm’)`
Matplotlib has a lot of different “”cmap”” styles to choose from, in their documentation.
You can make a ROOT.TProfile2D() profile plot first, then extract the entries into x, y, z triplets, and plot those as above.
Also see https://stackoverflow.com/questions/33942700/plotting-a-heat-map-from-three-lists-x-y-intensity for a bunch of different methods (thanks to Josh for that link!).”
Examples for using Pandas DataFrames
“This example is from DUNE, but it shows some of the things Pandas can do:
So a dataframe is just a collection of series. For instance the reset_data DataFrame, may include the Series: pixel_x, pixel_y, and reset_time. You can plot stuff by telling it what series to plot.
`reset_data.pixel_x` will give the pixel_x series for the whole sample
if you want to look at the pixel_x series but constrain it to only one event, you can say
`reset_data[reset_data.event == 0].pixel_x`
or in a range of events
`reset_data[reset_data.event >= 0][reset_data.event < 10].pixel_x`
You can also do multiple constraints
`reset_data[reset_data.event == 0][reset_data.pixel_y == 0].pixel_x`
Series has a builtin function `.tolist()` which you can use to explicitly turn a Series into a list. So using the most recent example:
`reset_data[reset_data.event == 0][reset_data.pixel_y == 0].pixel_x.tolist()`
But i dont think you need to do that to be able to plot them.
Lastly, this is a cool function I have used a lot in the past few years. It basically reads your dataframe and groups stuff by unique pixel-events (or whatever you tell it to group by). So like in this case pixel (1,2) in event 1 is a separate row from pixel (1,3) in event 1, as well as separate from pixel 1,2 in event 2. And it will tell you how many resets you have for each unique active pixel
`unique_active_pixels = resets.groupby([‘event’,’pixel_x’,’pixel_y’]).size().reset_index().rename(columns={0:”,1:’event’,2:’pixel_x’,3:’pixel_y’,4:’nResets’})`”
What to do if your Jupyter Notebook throws [Error] and Freezes
“If your Jupyter Notebook session puts up a brown [Error] box, and won’t let you save or checkpoint your notebook, leave it alone. Delete the Jupyter “”file browser”” tab (do not “”Logout”” or “”Quit””), then go back to the original launch (“”My Interactive Sessions””) and reconnect using the blue [Connect to Jupyter] button.
You should then be able to relaunch your notebook. If necessary, use copy-paste to transfer unsaved cells or edits from the broken [Error] session to the newly launched session. Once that’s done, you can delete the broken tab.”
Common Scripts
Custom emails in SLURM job
In order to send custom emails from inside a SBATCH environment, use the slurm-mail.sh script located in mitchcomp/bin. Documentation for usage is in the header of that script.
Other Common Issues
Use dos2unix after transferring text files from Windows to HPRC (Unix)
When you create or edit a text file on a Windows machine (NotePad, TEdit, whatever), it will have Windows end-of-line markup. If you transfer that file to Linux (HPRC or elsewhere), you need to convert it to having Unix end-of-line marks, or it will not be properly readable. Use the command `dos2unix` with the filename, and it will be converted in place.
Globus Upgrade Warnings can be Ignored
If you see the warning message “This endpoint must be migrated to version 5 by Dec 18, 2023” it can be safely ignored. According to HPRC, “Grace’s endpoint will be upgraded during November maintenance. Terra’s endpoint will also be upgraded before the December 18th deadline.”
Math or other “fancy” notation in Google Docs
“Google docs supports Unicode characters and Unicode “”combiners”” (like accents, diacritics, vowel markers for Arabic and Hebrew, etc.).
First type the base character (“”1″”). Then go to Insert -> Special characters and in the search box type “”combining overline””. Choose the single overline by clicking it. Close the box with the “”X”” in the corner, and you’ll see the result: 1̅
If you think the overline is too long (it’s designed to overlap with adjacent overlines, so you get one long bar), you can use “”combining macron””: 1̄
You can do the same with vector arrows, underline or double-underline markers (for tensors), hats, etc. The keyword “”combining”” will help you identify the characters that do this.