Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders. Subscribe Now
Data doesn’t just magically appear in the right place for enterprise analytics or AI, it has to be prepared and directed with data pipelines. That’s the domain of data engineering and it has long been one of the most thankless and tedious tasks that enterprises need to deal with.
Today, Google Cloud is taking direct aim at the tedium of data preparation with the launch of a series of AI agents. The new agents span the entire data lifecycle. The Data Engineering Agent in BigQuery automates complex pipeline creation through natural language commands. A Data Science Agent transforms notebooks into intelligent workspaces that can autonomously perform machine learning workflows. The enhanced Conversational Analytics Agent now includes a Code Interpreter that handles advanced Python analytics for business users.
“When I think about who is doing data engineering today, it’s not just engineers, data analysts, data scientists, every data persona complains about how hard it is to find data, how hard it is to wrangle data, how hard it is to get access to high quality data,”Yasmeen Ahmad, managing director, data cloud at Google Cloud, told VentureBeat. “Most of the workflows that we hear about from our users are 80% mired in those toilsome jobs around data wrangling, data, engineering and getting to good quality data they can work with.”
Targeting the data preparation bottleneck
Google built the Data Engineering Agent in BigQuery to create complex data pipelines through natural language prompts. Users can describe multi-step workflows and the agent handles the technical implementation. This includes ingesting data from cloud storage, applying transformations and performing quality checks.
The AI Impact Series Returns to San Francisco - August 5 The next phase of AI is here - are you ready? Join leaders from Block, GSK, and SAP for an exclusive look at how autonomous agents are reshaping enterprise workflows - from real-time decision-making to end-to-end automation. Secure your spot now - space is limited: https://bit.ly/3GuuPLF
The agent writes complex SQL and Python scripts automatically. It handles anomaly detection, schedules pipelines and troubleshoots failures. These tasks traditionally require significant engineering expertise and ongoing maintenance.
The agent breaks down natural language requests into multiple steps. First it understands the need to create connections to data sources. Then it creates appropriate table structures, loads data, identifies primary keys for joins, reasons over data quality issues and applies cleaning functions.
“Ordinarily, that entire workflow would have been writing a lot of complex code for a data engineer and building this complex pipeline and then managing and iterating that code over time,” Ahmad explained. “Now, with the data engineering agent, it can create new pipelines for natural language. It can modify existing pipelines. It can troubleshoot issues.”
... continue reading