GoKawiilInteractive parts of this page require JavaScript, but you currently have JavaScript disabled in your browser. Some parts of this page might not work.
Some time ago, I joined Deepnote as a software engineer. Deepnote makes a cloud notebook for data teams. I, however, didn't have any background in data. But I knew what a notebook was and I thought it would be interesting to work on this kind of project. I didn't think the data field was that far from software engineering, I always thought of them as adjacent fields.
Soon after joining I realized that I didn't know a thing about it! There are so many data tools besides notebooks, and I had no idea what they were used for or what the general work process in data science was. And if I don't know how various data tools are usually used, or how they interact with notebooks, I can't really suggest a good feature for a notebook or spot a problematic UI flow.
But after reading tons of articles, poking my colleagues and strangers on the internet with questions, studying workflows and struggles of our clients, and just giving myself some time to internalize this new knowledge, it did get better. Unfortunately, there wasn't a convenient "guide to data tools for software engineers who found themselves in a data company and have no idea what all those words mean", or I just couldn't find one.
I don't work on notebooks anymore, but data got me hooked, so my next job was in the field as well. This time I'm working on a tool for analytical type specialists rather than scientific (see the next chapter for an explanation!), but general knowledge about data tools and processes is still very useful! So I tried condensing it into an easy-to-read article and I hope it will help lost software engineers feel a bit more comfortable.
Who is this article for
As you can guess, I'm not really interested in becoming a data specialist per se. As such, this article won't cover, for example, how to create a dashboard in Metabase, basics of statistics, or how to administer a Spark cluster. It's aimed at developers who, for some reason, need to learn wtf all those people from the data team are talking about.
In this article, we will briefly go over the data lifecycle: where data comes from, how it's handled, how it's stored, and how it's displayed. You will understand to which stage each particular tool belongs and which tasks it solves for people working with data.
We won't go into details about setting up each tool or comparing tools inside the same class in-depth. Trust me, the article will be pretty lengthy as is, even without going into any of those details.
Flavors of data professions
... continue reading