Cloud file services startup LucidLink tripled in size over the past year, keeping remote media content workers fed fast with data through sub-file movement and caching, and it has zero egress costs with IBM COS and a low-cost cloud storage tier coming. We were briefed by CEO Peter Thompson and take a comprehensive look at his company and its technology in this two-part article.
LucidLink’s Filespaces product supplies remote workers’ laptop and desktop systems with file data sourced from an underlying cloud object store distributed in part-file chunks to speed data access. It synchronizes the file system state across the working teams’ systems using local metadata stores. Thompson said that the pandemic provided a boost to its sales and changed its development direction to match the new working-from-home style of many media creative and editing people.
Thompson told B&F: “On pretty much every metric that we track over the last year even during this challenging recession, and contractions going on and revaluations and everything else, we’ve managed to triple the business.”
LucidLink is a 2016 startup operating in the same general field as CTERA, Nasuni and Panzura but has been flying a little under the radar.
Co-founders Thompson and CTO/chairman George Dochev discovered that “collaborating on high velocity files through fast rates of change, especially when those files became big, became just harder and harder and harder. It was an unsolved problem.”
They tried “to figure out what is the problem, because there’s lots of bandwidth, latencies are reasonable, there was there’s just kind of no technical reason that we could see that made this such a painful process when you try to access files over distance.”
“So we threw everything we knew out the window, and began looking at how can you access files over a long, thin connection. And that’s really what we’ve done: we’ve made a kind of a cloud-optimized file system, that promotes that type of access.”
The primary use case they saw was “pattern matching for distributed teams who collaborated on large sets of data, and specifically on big files. And this crops up in a lot of different areas: in video creation and creative content; in the AEC, space for architectural design and CAD CAM files; in medical imaging; in life sciences research; in geospatial models in the oil and gas industry. In all kinds of places this pops out.”
It’s also active in sports broadcasting and corporate marketing departments for video production. This is its fastest growing business segment.
Product and pandemic
The first product was built by 2020 and LucidLink was beta testing it in these vertical markets when COVID-19 struck, unexpectedly ramping up interest in its technology. Thompson said: “It was at the beginning stages of the pandemic when suddenly, we began receiving inquiries from media teams from broadcast people.”
Thompson said LucidLink was told: “We sent our editors home to work from there, gave them a laptop, a VPN connection and a hard drive. And that’s sort of getting us through the next couple of weeks, but then we don’t know what we’re gonna do.”
These customers “would try LucidLink’s Filespaces, usually on a demo, and it would just roll directly into production.”
This just snowballed. “We began having large customers – you know, kind of in that $100,000 range customers – who began immediately using us in production for distributed teams across the country, or even across the globe. And we grew our business from 2020. Boy, it grew so fast. We were so small at that time, you know, it was maybe a 10x, or 20x, or something kind of growth to 2021.”
This validated LucidLink’s approach of concentrating on large files. Thompson said: “We knew that the larger the file was, the more distributed the team, the quicker they would see the magic.”
If files were small and datasets not that big, as with sharing office files and things like PDFs, then file sync and share was developed for that back in the day and is fine. Customers aren’t going to notice that much of a difference from it with LucidLink’s FileSpaces.
FileSpaces comes into its own with larger files and datasets. “LucidLink is giving you the entire set of data. It’s allowing you to mount your cloud drive as if it’s local, and the application and the user don’t have any idea where the files are actually located.”
What Lucid’s tech is not doing is having a user “click on the file, download the entire thing, edit it and push it all back up to the cloud, and then synchronize that across to everywhere else.” Instead “you’re actually using it in place. Or, more accurately, we’re distributing the file, we’re downloading portions of it, you’re using your active set.”
“Through the way that we’ve done this distributed file system, the source of truth remains in the cloud, but you’re able to use the file as if it’s local. And more importantly, you’re able to collaborate with other users who are distributed around in different areas.”
No workflow changes
What this meant for the media editors was that there was no change. “I think our success was driven by those editors and the users. They really embraced this because they didn’t have to change any of their workflow. They didn’t have to learn new tools, they had the muscle memory that was intact, and in place. They used to be in the office, connected to the LAN, going to the NAS or the file server. Now they’re at home, connected to the internet, going to the Lucid mount. And there was no difference.”
“The performance was so great that they just forgot about it. We would just fade into the background, which is exactly what we wanted to do. We wanted to disappear. The more we could disappear, and have them back in their workflow and on their tools, without even thinking about us, the more we knew we’d be successful.”
Lucid changed its hiring pattern. “We began hiring less storage and networking specialists, and more workflow specialists – people who knew that knew the tool sets and the workflow for for creative content, and edit and video editing.”
This fit the work-from-home scene and the large files and datasets market meant LucidLink kept on growing. “It hasn’t shown any signs of showing slowing down. In fact, the velocity is still increasing. I think the reason for this is that it has shown to be such a high productivity increase for these teams. To put it quite simply, they’re no longer waiting for files to download.”
Metadata streaming
However, the increased size of the business caused a technology rethink. “We realized that we were going to hit some scalability issues at some point. And we knew that we needed to be able to support a billion files and tens of thousands of users. The initial product wasn’t quite going to do that, at least not comfortably.”
“In late 2021, through to mid-2022, we reworked our platform. This is what we we termed Lucid 2.0. And the intent of this was to increase performance, but more importantly, to allow for this kind of massive scale in numbers of files that we could support and numbers of users that could be in the system.”
This involved introducing metadata streaming. “Because these file systems were getting so large, the metadata associated with them was becoming a big data problem in and of itself. And then we said, alright, we can’t continue going on synchronizing metadata across tens of thousands of clients – that’s not efficient.”
“So we did the same type of streaming technology that we did with the file data for the metadata. That’s what we call our metadata streaming. This means that, even if someone joins fresh on a very large file space, they’ll have immediate access to it.”
After the LucidInk 2.0 platform change “we began focusing our efforts on specific user requests. And we began rolling out some features,” Thompson said.
One was single sign-on (SSO) and a second was a a remote upload indicator to say when a file is is fully loaded and ready to go. Another one is full path shares: “The ability to show a path to your assets, within your tools consistently across different sets of tools, and different platforms, and different operating systems.”
A fourth is a public API so that Filespaces can integrate with infrastructure tools such as Adobe Premiere or Autodesk Flame. If customers need to go to a drive, a mount or a NAS, they can go to Filespaces instead, which shows up as a drive. Thompson said: “We can be that centralized, consistent repository that spans across all of those tools, and therefore that entire workflow. The amount of pushing data from one point into another during that life cycle is dramatically reduced.”
The integration with media asset management systems is also improved because of this API. Other roadmap items include a web client and a mobile device experience.
***
Part two of this account of LucidLink and its technology will look at IBM COS, egress charges, direct object access and LucidLink’s competition.
