GoKawiilOne of my projects is predicting the spectra of proteins. To do this, I design molecules, then get a computer to optimise their geometries and find the energy levels. Most do this kind of work on a University high performance computing cluster.
But here's something I noticed when doing this many times. When I need to remote into a cluster, the first part is to log in. I need to first login to the University VPN, which means using some outdated software to get the VPN set up from my computer. And then there is the authentication part, where the VPN software makes you jump through multiple authentication and 2FA processes to log in. And then when the 2FA window finally comes, I need to dig out my phone from somewhere in my room to get the code. Probably on average I find that the VPN + ssh + 2FA process takes 30 seconds every time I need to do something, which generally feels like too much.
It's not too much work overall, but there are a lot of intermediate steps and riff-raff.
I call this clunkiness. Something is clunky when there are many intermediate steps required to do one task. You know those really annoying shower controls in fancy hotels when you need to adjust two separate knobs for the pressure and temperature separately? That's clunky.
It’s clunky to do work in University compute clusters.
A bit of clunkiness in using something is usually fine. But if the clunkiness is even slightly more than a bit, the effects can be disastrous. If there are too many intermediate steps to get started to work on something, I become less motivated to work on it, forget about it, and in the worst case scenario just drop the endeavour entirely. Which means that an entire stream of work can be lost for small and trivial reasons. This happens with a lot of things! For example, it's really clunky to descale my coffee machine, and so I never end up doing it, and so it always breaks.
The friction in using these clusters has been enough to actually stop me from doing work that I would've done otherwise. If I was a full-time PhD student I would probably muscle through each awkward step to login. But I'm not, I work on this independently and am short of time. I am also mentally tired since I do this after work. And these things make it such that access to the compute needs to be instant and no-faff for me to justify getting started. This is something independent scientists try to optimise, which I wrote about in one of my older posts.
In terms of background, many individuals did research whilst they were in university, but had to give it up as they started their jobs. There were also individuals who worked as engineers in top tech companies, science adjacent companies, or start-up founders, who already had high level experience in project management and research. Some had already submitted papers! Most tried to do research in their free time out of pure passion, and chose day jobs in industry to support financial and lifestyle goals.
Source: What I Learned from Speaking to 500 Independent Scientists
There is also a layer of friction between doing the computations and then analysing the logs that comp chemistry programs spit out. If you want to visualise orbitals or look at the outputs of your simulations, it's really awkward to run Python whilst being in the cluster, and the cluster has no GUI display. The way I used to do this was by doing SCP and copying the log files back and forth. Efficient I know... I used to use MacFUSE with sshfs but it turned out to be really unreliable as well, and quite slow. I wrote about the procedure a few years ago.
... continue reading