AWS has made it cheaper to search large volumes of log data by inserting an ‘UltraWarm’ storage tier between cheap, slow S3 and fast, expensive Elastic Block Store (EBS).
Using the open source Elasticsearch with UltraWarm is one-tenth the cost of other options, according to AWS. It does not say what those other options are – but it is safe to say it costs more to use ElasticSearch with EBS, which puts log data in Elastic Block Store (EBS) volumes attached to each Elasticsearch node.
Raju Gulabani, AWS VP of databases and analytics, said in a statement: “Our customers tell us that log data offers a wealth of operational and security insights, but that the storage of log data quickly adds up, and proves cost-prohibitive over the medium and long term. UltraWarm is the most cost-effective Elasticsearch-compatible storage solution available. It is also performance-optimised, so customers can investigate and interactively visualise their data while they embrace data at scale.”
Log data in EBS is classified as hot data and copied to replicas to ensure its durability. EBS space is also reserved for Linux and for Elasticsearch. The log data is organised into shards – indices of documents – and a primary shard is the main index. A 10GiB primary shard takes up about 26GiB of EBS storage, due to the overhead that AWS requires. Elasticsearch customers pay for the entire space.
S3 storage provides durability, removing the need for replicas, and abstracts EBS-required operating system or service considerations. But it can only be searched slowly, whereas the UltraWarm tier is searchable by Elasticsearch and “provides the type of snappy, interactive experience that Elasticsearch customers expect”.
Mix and match
UltraWarm is a distributed cache layered above S3. It is populated with frequently accessed blocks of data from S3 and placement algorithms are used to identify less frequently accessed blocks in the cache and shunt them back to S3.
You pay for the UltraWarm storage you use. For example, “An ultrawarm1.large.elasticsearch instance can address up to 20 TiB of storage on S3, but if you store only 1 TiB of data, you’re only billed for 1 TiB of data.”
Customers pay an hourly rate for the storage provisioned and an hourly rate for each UltraWarm node. Pricing details can be found on the Elasticsearch website pricing page.
AWS suggests you use a hot EBS tier for indexing, updating, and getting the fastest access to data. The UltraWarm tier would be used for less frequently accessed data but where a fast search response is still needed. So, the customer could put current data in EBS and historical data in the UltraWarm tier and access both tiers, using the Elasticsearch Kibana interface.
UltraWarm supports Elasticsearch application programming interfaces (APIs), tools, and features, including enterprise-grade security with fine-grained access control, encryption at rest and in flight, integrated alerting and SQL querying.
UltraWarm is available on Amazon Elasticsearch version 6.8 and above in US East (N. Virginia, Ohio), US West (Oregon, N. California), AWS GovCloud (US-Gov-East, US-Gov-West), Canada (Central), South America (Sao Paulo), EU (Ireland, London, Frankfurt, Paris, Stockholm), Asia Pacific (Singapore, Sydney, Tokyo, Seoul, Mumbai, Hong Kong), China (Beijing, Ningxia), and Middle East (Bahrain), with additional regions coming soon.
