.webp)
Fullscript provides an online prescribing platform that allows practitioners to create personalized plans for their patients, with access to over 20,000 practitioner-grade products from 300+ brands.
https://fullscript.com/
This is the story of how dbt Cloud and Secoda helped Fullscript scale its data workflow and modernize its data stack, while in the middle of a company merger.
This case study was prepared in proud partnership with dbt
By the numbers:
Fullscript® is the leading online platform for practitioners to manage and dispense healthcare’s best supplements — from their clinic or directly to patients' doors.
Practitioners write online supplement plans where patients order wellness products and find resources to improve their health education. Practitioners control what their patients pay so they can grow their practices, offer deep discounts, or strike a comfortable balance of both. Fullscript’s US catalog also supports wholesale orders for in-office dispensing.
The aim: Modernize a legacy data stack, and integrate new data sources following a major acquisition.
The challenge: Quickly upskilling team members with limited data experience and integrating a huge number of new data sources.
Founded in Ottawa, Canada, in 2011, Fullscript works to help more than 90,000 health practitioners deliver better care to their patients through its online prescribing platform. This service allows practitioners to create personalized plans for their patients, with access to over 20,000 practitioner-grade products from over 300 brands.
Any operation of this scale generates a vast amount of information, and Fullscript is no exception.
“Data is fundamental to Fullscript,” said Amit Jain, Data Team Technical Director. “We completely rely on it to report on both the financial and operational aspects of our business.”
The team tracks customer engagement through clickstream data on its front-end applications and the back-end operations for warehousing and shipping. This boosts customer service and allows the business to make decisions based on accurate, reliable numbers.
In Fullscript's growth journey through 2021, the organization structure presented challenges when it came to effective collaboration between data engineering and data analysts, who used distinct practices.
“They were inclined to use software engineering development practices, which meant they didn't speak the same language as the data analysts,” said Amit.
With the progression of time, the initial team members who had an in-depth understanding of the legacy data engineering systems underwent role transitions, leaving the data warehouse team devoid of crucial historical knowledge.
Fullscript’s legacy stack consisted of Postgres, SQL and Python scripts embedded in Argo - a Kubernetes-based orchestration tool. However, it soon became clear that the organization needed to update these systems to a more accessible, reliable data stack in order to scale.
“From a data analytics point of view, our legacy systems were tough to manage when things went wrong, and the data didn’t match between source and target,” explained Jain. “We couldn’t identify faulty sources to apply the appropriate fixes.”
The fact that all of Fullscript’s tools were custom-developed further complicated matters.
“We were almost at a standstill,” said Jain. “The data pipeline was written using complex orchestration mechanisms and had SQL and Python wrapped inside the code. Our team was spending all its time just making things run.”
This massive drain of time and resources meant the team didn’t have the capacity to improve the systems or build any new products. Fullscript wanted to bring in data from their CRM, NetSuite ERP, Google Analytics, and Facebook Ads, but there was no time to add new sources.
“Our teams were at capacity just managing issues as they came in. We didn’t have time to add new requirements or sources.”
Jain and his team knew that they needed to modernize their stack. They needed to find reliable, scalable, accessible tools to transform and catalog their data.
“We estimated that our modernization project would take almost nine months,” Jain explained. “We wanted a way to track how the migration was taking place because we had to have both systems running in parallel.”
After researching and experimenting with several potential tools, the Fullscript team decided that dbt Cloud was the best candidate for data transformation, while Secoda —an AI-powered data search, cataloging, lineage, and documentation platform—could help them keep track of information as it moved through their systems.
According to Jain, “Fullscript chose dbt—and dbt Cloud in particular—because it offers excellent integration of data engineering and analytics. It’s also very scalable, has great data analytics capabilities, and is easy to adopt and ramp up quickly.”
“At the same time, we found Secoda was the most flexible cataloging tool available. We looked at many other options, but there were always some gaps or source types they didn't support. I can’t imagine anyone running a modern data stack without Secoda.”
Soon, the team was able to run a proof-of-concept project during one of the company’s regular hackathons. Almost immediately, they delivered impressive results.
“I worked with a few data analysts on our team and used dbt Cloud to create a report within a couple of days,” said Jain. “In the old setup, where everything was custom developed, this would have taken months.”
Transforming an organization's data stack isn’t a small or simple project. That’s why Fullstack’s team took advantage of dbt’s ability to easily scale up as required.
“You could start with the open source version and then explore options to add dbt Cloud and then dbt Enterprise for the added support,” said Jain.
“It was a very flexible model. You could easily start small and prove your concept; this was especially useful when getting executive buy-in.”
With no engineers left from the team that had designed Fullstack’s legacy tools, the business needed a tool that was easy to onboard and adopt.
“That's what we found with dbt,” said Jain. “Using just SQL, somebody with a data analytics background could use the tool fairly easily.
“For the initial setup, much of the tools just ran out of the box.”
Clear documentation and accessible support are vital in getting projects off the ground. The Fullscript team found dbt Cloud could deliver on both fronts when modernizing their data stack.
“We could use the dbt documentation to create a reference architecture for the initial implementation,” said Jain. “From there, we were able to work with the dbt enterprise team to refine it according to our needs.
“The support and training capabilities dbt Cloud offered were almost as important as the tool itself.”
While it’s easy to think of ease-of-use and well-designed support packages as fringe benefits, the combination of features allowed dbt to deliver when it came to Fullscript’s bottom line.
Jain noted that even though the business had moved from free software to a paid product with dbt Cloud, the switch still helped save on employee costs.
“A Kubernetes-based solution required a lot of engineer support and a wide array of specialized engineers,” he explained. “Once we moved to dbt Cloud, we didn't need engineers with very high Docker and Kubernetes expertise. Knowledge of SQL and analytics engineering capabilities were enough to use the platform.”
The flexibility of dbt Cloud, Secoda, and Fullscript’s new data stack was tested mid-implementation. The team was only a few months into the migration when the business merged with Natural Partners—a supplements provider operating on roughly the same scale as Fullscript.
“Overnight, our data sources doubled,” explained Jain, “The acquisition plans had been under wraps, so teams weren't even aware it was coming.”
Fortunately, the Fullscript team had already started making full use of Secoda, which was able to help them integrate the metadata from Natural Partners’ data warehouse.
“Secoda was very helpful,” said Jain. “We were able to quickly connect to these new sources and bring in metadata from their new source databases.”
While the acquisition represented a significant challenge for the data team, it also provided an opportunity to showcase the power of their modern data stack. Leveraging dbt Cloud, the team could quickly extend its existing data model and effectively map the data.
“Mapping the data took a lot of upfront work, but once it was done, we could extend our existing models,” said Jain. “All of that was powered by dbt Cloud, and it proved the scalability of the modern data stack. We did it quickly and finished the migration within the planned nine-month period.”
“I can't imagine how we would have been able to scale so easily or double in size without dbt Cloud and Secoda. I know we made the right choice with this set of tooling, and if I were to go back in time, I would choose it again.”
With a modern, powerful data stack now established, Fullscript plans to continue building its capabilities over the coming months and years.
One of the key goals is to extend its data models to support the new requirements expected to come in from across the business. One key example is the ability to support more Artificial Intelligence Markup Language (AIML) use cases, anticipating increased use of AI across both Fullscript and the broader data ecosystem.
“Mostly, we’d like to use AI/ML around the recommendations engine,” explained Jain. “Specifically, for product recommendations using large language models, ChatGPT-style.”
“For example, the ability to develop guidelines for product recommendations would greatly help our practitioners. Something like that would have a major impact on their day-to-day.”
© 2024 Secoda, Inc. All rights reserved.
By using this website, you accept our Terms of Use and Privacy Policy.
