We sat down with Lindsay Murphy of Secoda, Scott Breitenother of Brooklyn Data Co., and Julie Beynon of Census to cover sh*t they wish they did in 2023, lessons learned, and what they will carry forward with their data teams in 2024.
Table of Contents:
- 2023: The Year of Transformation
- Cost Monitoring and Spending on Data Teams
- Data Democratization and Governance
- Quality and Value Demonstration
- 2024: Anticipating the Future
- Innovations Shaping Workflows
- Reflecting on Challenges
2023: The Year of Transformation
Acceleration and Maturity: The data industry has reached new heights of acceleration and maturity, fostering unprecedented innovation that propels us into a transformative era.
Industry Events: Gatherings like the Snowflake Summit and Databricks Coalesce Conference have become vital for the data community, shaping the trajectory of the industry through idea exchange and innovation. We’ve also seen more local meetups in cities, clearly there is a desire from the community to gather in person to exchange ideas.
Evolution of Audience Sophistication: Observing the industry's sophistication evolve is enlightening; as the landscape matures, professionals adeptly navigate complex data implementations.
Growth of Data Platforms: We’ve seen huge growth for data platforms, evolving into dynamic hubs, and providing transformative insights beyond mere storage. Platforms are aiming to become swiss army knives for data teams, and we’re seeing more and more features being added to already beloved platforms.
Cost Monitoring and Spending on Data Teams
General Trend in Data Spending: A panoramic view of data spending unveils an upward trend, underscoring data's strategic importance - organizations invest strategically to harness its potential. We’ve also noted in our previous articles about the costs of implementing a modern data stack, and companies were feeling it in 2023.
Consumption-Based Cloud Spend: The paradigm shift towards consumption-based cloud spending marks a new era, emphasizing dynamic, on-demand utilization for increased efficiency. The introduction of new monitoring and observability tools to help data teams keep tabs on their spending were also on the rise.
Thoughtful Spending: Amid growth, the emphasis on thoughtful spending and ROI considerations is paramount—strategic impact over sheer investment size. We saw a lot more conversation spurring up around how data teams can become revenue generators, rather than cost centers as the price of data management rises.
Investing in Critical Roles: Redirecting investments towards pivotal roles like analytics engineers, architects, platform engineers, and data governance specialists ensures a comprehensive approach to data management. We’re also seeing a trend in specialized analysts, rather than generalists (think a marketing analyst or product analyst vs. a general data analyst). A notable shift from generic titles to specialized roles reflects the industry's need for expertise in addressing modern data challenges.
As the industry matures, there's a rising demand for specialized skills, emphasizing in-depth, domain-specific knowledge over one-size-fits-all approaches.
Data Democratization and Governance
Emphasis on Democratization: The pursuit of efficiency brings a growing emphasis on simplifying data workflows and structures, making data more accessible to all employees at an organization. We’ve begun to ask ourselves big questions about the data culture that exists at our organizations, and how we can build and intentionally shape the culture as we move forward.
Importance of Data Governance: Data governance and guardrails play a pivotal role in ensuring data integrity and security, creating a delicate balance in the data ecosystem.
Tools that combat data sprawl, such as data management platforms and catalogs, become essential for understanding and optimizing data assets. Robust metadata quality management emerges as a priority for the success of generative AI tools, enhancing data discovery and governance.
Role of Analytics Engineering: Analytics engineering emerges as a linchpin in simplifying complexity, bridging the gap between raw data and actionable insights.
Quality and Value Demonstration
The Modern Data Stack: The realization dawns that as we've conquered old problems, new challenges have surfaced, particularly in scaling and maintaining efficient data teams using the modern data stack.
The core challenge lies in the balance of expectations—data teams yearn for speed, quality, and cost-effectiveness, echoing the classic dilemma: "You can have it good, fast, or cheap—pick any two."
Getting Ahead of Issues: The industry's proactive stance on data quality ensures a smoother data journey by anticipating and preventing issues - this is good news for data team’s as the value of their work is being recognized by the collective, but this also means there might be more of a magnifying glass on their work they are doing.
Demonstrating Value: External pressures compel data teams to move beyond tasks and showcase value through high-impact initiatives, aligning with organizational goals.
Monitoring the performance of both the team and the tech stack is vital, with established KPIs shedding light on strengths and weaknesses. Enforcing cost containment methods from the inception of the modern data stack ensures long-term financial resilience.
2024: Anticipating the Future
The road ahead is filled with challenges and opportunities; anticipating 2024's shifts is crucial for staying at the forefront of the data revolution.
Thoughtful Data Collection: The future demands a thoughtful approach to data collection and usage, prioritizing quality, ethics, and sustainability. Striking a delicate balance between harnessing valuable insights and respecting individual privacy becomes paramount, steering the data industry towards a more responsible and sustainable ecosystem
Importance of Self-Awareness: Self-awareness within data teams emerges as a critical component, empowering continuous adaptation and improvement. This involves cultivating a deep understanding of the team's own processes, strengths, and areas for enhancement. By fostering a culture of introspection and analysis, data teams can proactively identify bottlenecks, streamline workflows, and enhance overall efficiency.
Real-Time Data Transformation: The palpable excitement around real-time data transformation unlocks new possibilities, enabling organizations to respond dynamically to data insights.
Warehouse as a Data Platform: Data warehouses transform into dynamic data platforms, propelling personalized experiences and insights beyond conventional storage.
Innovations Shaping Workflows
Productivity Tools: The shift in perspectives on tools like ChatGPT and generative tools as productivity aids marks a transformative trend, akin to the impact of early internet tools.
Applications in Data Teams: Generative AI tools assist data teams, introducing new roles like prompt engineer and AI engineer. Applications of AI for data teams are evolving rapidly, revolutionizing data processing and decision-making. AI tools, such as generative models and machine learning algorithms, empower data teams to automate complex tasks, enhance predictive analytics, and extract meaningful insights from vast datasets.
Reflecting on Challenges
Enrichment Challenges: We discussed the challenges in solving data enrichment and revealed the ongoing struggle to balance cost-effectiveness and real-time enrichment.
Incomplete Enrichment Solution: Acknowledging the importance of enrichment and the lack of a comprehensive solution, the focus shifts to specific use cases for effective enrichment.
Conclusion
2023 marked a transformative year in the data industry. Accelerated maturation, strategic investments, and a focus on specialized roles demonstrated the industry's adaptability. Anticipating 2024, the focus on thoughtful data collection, self-awareness, and exciting trends like real-time data transformation and AI applications signals a promising future. Challenges persist, notably in cost containment.
We can’t wait for 2024 - we’re excited to keep hosting these important conversations with experts and sharing their insights with you. Stay tuned for more articles, webinars and resources for you and your data team!
