At Texas A&M University, we have access to a high performance computing cluster. All of the software needed will be available on HPRC, so to save time and effort, we encourage you to get an HPRC account. The following onboarding instructions will be for users on HPRC. There are still plenty of ways to do this locally on your personal PC, but you will have to read the instructions and adapt them for your own setup.
Getting an HPRC account
In order to get started working with QPix, you will need to have access to HPRC. In order to do that, you need to be part of MITCHCOMP¹. You can find the step-by-step of how to get an account by going to Getting an HPRC Account. These instructions only work if you are a mitchcomp user, so make sure you are.
¹If you are in TobackGroup, but are not sure whether you are in mitchcomp, you can ask, or just to be safe, you can follow the instructions for new users here.
Setting up your HPRC account
Once you have access to an HPRC account, it is time to set it up to be able to use the QPix software. This will include installing the relevant modules, setting up an environment, and installing the QPix software.
It is recommended to do the onboarding on the Grace cluster as it’s faster and more powerful.
Dependencies
The QPix software has multiple dependencies that must be taken care of before attempting to install. These dependencies include: Python3, ROOT, Geant4, CMake, make, numpy, matplotlib and uproot. Thankfully most of these are already installed on the Grace cluster as modules, and can be added to your .bash_profile so you never have to worry about building them again.
Access your .bash_profile by running
$ cd
$ vim .bash_profile
Once in .bash_profile, you should edit it so that the entire file reads²:
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export PATH=$PATH:$HOME/.local/bin:$HOME/bin:/scratch/group/mitchcomp/bin
source module-setup
# conda deactivate
#Desired build environment
## Using foss-2020b toolchain, with GCC/10.2.0
module add foss/2020b
module load buildenv/default
module add CMake/3.18.4
module add git/2.28.0-nodocs
²It is important to note that we use a completely different toolchain for DUNE/QPix than we do for anything else Toback Group related. This may conflict with other setups you already have.
By changing this, you ensure that every time you open a new terminal, you are set up on the foss-2020b toolchain, and have access to CMake, make, GCC, and git. It is important to note that if you are trying to do work for both DUNE/QPix and CDMS, these two groups use software on different toolchains and you will need to switch between the two in order to use them.
For the rest of the dependencies, a DUNE-setup script is already created, which will take care of the rest. It is located in /group/mitchcomp/bin/ and you should be able to access it simply by running the following command in your home directory.
$ ln -s /scratch/group/mitchcomp
$ source qpix-setup
The above is recommended as it created a symlink from your home directory to the mitchcomp directory. This makes navigating between directories much easier.
If that doesn’t seem to work, you can also try:
$ source /scratch/group/mitchcomp/bin/qpix-setup
It is also important to note that the newer versions of ROOT (including the one that we use for DUNE/qpix) use a web browser for root TBrowsers. This will cause a problem for us when using HPRC. To avoid this, you will need to create a new file in your home directory
$ vim .rootrc
Enter the following line before saving and closing the file:
Browser.Name: TRootBrowser
You will be working primarily in your scratch directory as it has more storage than your home directory.
If you are new to HPRC, you can make a symlink to your personal scratch area from your home directory with the commands:
$ cd // moves you back to your home directory $ ln -s /scratch/user/[INSERT YOUR USERNAME] ~/scratch // creates a link called "scratch" in your home directory to your scratch area
Mitchcomp Github
Join the Mitchcomp Github and link your Github public key so you can access repositories shared in the Mitchcomp group.
1.) Join the Mitchcomp Github Organization.
Tutorial: https://mitchcomp.physics.tamu.edu/general-guides/github/
2.) Link a Public Key for the Github Organization.
At this point, your HPRC account has been set up to use the centralized Q-Pix software.
Below you will find information on our recommended analysis tools, HPRC/Computing Resources, and Q-Pix group information.
If you are going to be doing Q-Pix development work, proceed to the Developer Onboarding for further instructions.
Analysis Tools
Most of what you will be doing will require coding, both in simulations and analysis. Python (specifically python 3.8) is what is most commonly used for analysis and it will be good to have familiarity with it (You will also find a lot of C++ in the simulation code, so being able to at least read it would be good). In order to make sure everyone has the same basic skills, take some time to look at the resources below.
Jupyter Notebook
Jupyter Notebook is going to be what we use to contain all of our analysis coding. This comes as part of the anaconda package that you have included in your HPRC account setup. Once you have finished setting up your account, you should be able to access it anytime from the terminal, however HPRC has made it even easier to use Jupyter Notebook with its OnDemand Portal.
In order to access the HPRC OnDemand Portal, you must either be a) on campus, or b) using the TAMU VPN. It is guaranteed you will be doing some of your work at home at some point, the VPN is necessary, and if you don’t already have it set up, you can do so by going to https://it.tamu.edu/services/network-and-internet-access/virtual-private-networks/virtual-private-network-vpn/index.php and following the instructions.
Once you are logged into the VPN, you will be able to access the OnDemand Portal by going to https://portal.hprc.tamu.edu/. Assuming you have done the above onboarding on Grace (it is faster and more powerful), so you would obviously click the link for the “Grace OnDemand Portal”. Log in if you need to, and once you get to the home page, click the “Interactive Apps” tab in the top menu, and select “Jupyter Notebook” from the drop-down list. It will then bring you to a page which will help you set up your interactive session. To create a session compatible with your QPix software use the following settings:
Type of environment: “Module load + Python virtualenv
Module Selected: “qpix-jupyter/1.1-Python-3.8.6 (foss/2020b)”
Optional Environment to be activated: [leave empty]
Number of hours: [enter number of hours you will be working for]
Number of cores: 1 [unless you will be running something that needs multiprocessing]
Total Memory: [How much memory (in GB) you will need. To start, 7 GB is plenty (and is the minimum)]
Node Type: ANY
Account [leave empty]
Email: [leave empty]
It is important to note that when you get an HPRC account, you are allotted a specific number of core-hours (like a budget). Any time you make a session or run a job, you use core hours equivalent to the number of cores times the number of hours time half the memory (in GB) you are requesting. This is to prevent unnecessary waste of computing resources. So while asking for 25 cores for 10 hours and 100GB will definitely get your job done and be enough for you to accomplish work, you will be wasting you allotted core-hours, and 1 core for 10 hours and 4GB may still be plenty and will save your hours, while also keeping computing resources available for other users at the same time.
If a new session fails to start, make sure to include your account number. This can be found by running ‘myproject’ in the ssh environment on MobaXterm.
Python, numpy, matplotlib, and pandas
In order to do QPix analysis, we will be working with big data, which here we are defining as dataframes with hundreds of thousands of rows to millions of rows (free resume terminology for you). In order to work with this data, data analysis tools in python will be your best option. In the years I have worked with QPix, I have worked specifically with the numpy, scipy, pandas and matplotlib packages in python. While we don’t have specific tutorials for using each in QPix, we can point you to some good resources, however as physicists much of what we learn will be from Google when we need something new.
Most of the basic tools needed to perform data analysis for QPix will be discussed in the two LinkedIn Learning links under the “Analysis” tab on the groups onboarding page.
HPRC/Computing Tutorials
HPRC provides training modules, tutorials, and workshops for beginner, intermediate, and advanced users. More information on HPRC training can be found here.
Mitchcomp has computing resources including Python, C++, bash/linux, and HTML/CSS. Those tutorials can be found here.
Group Details
There are 3 group meetings that pertain to Q-Pix: SAQ Meetings, QPix Group Meetings, and QPix Simulation Meetings. To be added to the email list about the following meetings, email Jonathan Asaadi of UTA (jonathan.asaadi@uta.edu). Details are as follows:
SAQ Meetings
Usually held Thursdays 10:30am-11:30am. SAQ stands for Simple Analog QPix. These meetings are to discuss the progress of building a simplified version of the QPix with affordable and less-than-state-of-the-art parts while we are waiting for the real QPix to be built. They also discuss any tests and experiments using the SAQ.
QPix Group Meetings
Usually held every other Tuesday from 1pm-2pm. These are the full QPix meetings with the entire group. All relevant hardware and software can be discussed at these meetings.
Zoom Room: 555 242 4976 (pw: qpix)
QPix Simulation Meetings
Usually held every other Friday from 9am-10am. These are the smaller QPix meetings focused specifically on the software and the simulations.
Zoom Room: 928 3590 4093 (pw: 607861)