To win visibility – and ultimately insights – in the corporate -Cloud -Assets is becoming increasingly difficult.
Cloud Estates are extensive and fragmented, and the inventory functions in existing tools might be narrow and unintended and separate elements corresponding to cost and safety data into separate platforms with limited flexibility.
Cloud Governance Company Cloud query Positions itself to resolve this problem by centralizing cloud assets, security metadata and costs in a single place and it’s accessible through easy, integrated SQL queries and reports. The company follows a developer approach for cloud governance and draws data from over 60 sources inlay AWS, GCP, Azure, Okta and Wiz-in a single, requested data warehouse.
The company now pronounces a round of finance led by Partech in the quantity of $ 16 million so as to further scale its approach to cloud visibility.
“The biggest challenge for existing tools is that -for security, one for the prices, one for the inventory of assets -that make it difficult to get a uniform view of domains,” Cq founder Yevgeny Pats told Venturebeat. “Even easy questions corresponding to' Which EBS volume is attached to an EC2 switched off? Are difficult to reply without sewing several tools together.”
Cloud query under the bonnet
CloudQuery uses two key technologies under the hood: Data Warehouse and Open Source Databo Clickhouse and the Apache-Arrow framework for the event of knowledge evaluation applications.
This integrated high-performance plug-in architecture, which is integrated in GO, places APIS corresponding to AWS, Azure, Google Cloud Platform (GCP) and lots of other platforms that draw configuration, safety and price metadata. The platform repeatedly synchronizes data from dozens of cloud providers and services right into a normalized, central asset inventory.
“We give attention to data accuracy and freshness and synchronize with high frequency to make sure that teams work with essentially the most reliable and latest information,” said Pats.
He explained that the information are structured relative to the SQL Engine and integrated reports from CloudQuery, in order that teams can have full flexibility without counting on black box tools.
The company also uses “selective” large language models (LLMS) for query within the natural language, the SQL generation and the recommendations, “but all the time a couple of basis for exact, transparent data,” said Pats. He identified that tools corresponding to Claude and Openai, because KI SQL understands well, can create customer -specific reports and analyzes in easy English.
A developer approach is crucial, said Pats, since developers are ultimately the those that construct, operate and secure today's cloud infrastructure. Nevertheless, many tools were developed for the visibility of clouds for top-down governance, not for people within the trenches.
“If you utilize developers with accessible data, flexible APIs and mother tongue corresponding to SQL, enable them to maneuver faster, catch problems earlier and construct more safely,” he said.
Customers can find paths to make use of CloudQuery beyond the inventory of asset. “Many then start with visibility after which quickly grow into applications corresponding to compliance monitoring, security management, cost optimization and the identical core platform,” said Pats.
How Hexagon created a serverless data Lake for all cloud stores
An organization that already sees results is hexagon. The software company (Cloud Center of Excellence )'s Cloud Center of Excellence (Cocee) aimed to create a completely serverless data lake that might collect data from all cloud accounts and store it in a single data lake.
They also wanted to examine this data using SQL to visualise them with familiar tools (corresponding to AWS Quicksight) and to look at the history of their cloud configuration over time.
The team created a serverless data pipeline with CloudQuery to gather data from all accounts and store them in S3. AWS adhesive then adds data to a format that may query Amazon Athena, which Athena does, into the adhesive DB and visualized in QuickSty.
“Having a completely serverless solution was a very important requirement” Blog post. “This decision brought many benefits because no time -consuming updates and practically no maintenance are required.”
They needed to take care of some challenges, especially with Amazon S3 Support plugins. The CCOE team was one in every of the primary to try Cloudquery functions on the S3 goal, and offered knowledge that led to recent functions. This includes:
- Parket -Support: The CloudQuery file goal initially only supported CSV and JSON data formats. Errors in JSON interpretations prompted CloudQuery so as to add the support of parqueters.
- Data partitioning: A Cloudquery file goal -Plugin now enables partitioning on the primary letter (previously not available, which results in additional unnecessary steps).
- Resource view for Athena: CloudQuery initially only offered a resource view for AWS which are compatible with postgrres. However, Athena didn’t support this, so CloudQuery has added a function with which an inventory of all tables might be called as much as create or update a resource view.
The Figuereedo team used CloudQuery to exchange AWSS VPC IP Address Manager (IPAM) – which he described as expensive and limited since it doesn’t cover some other cloud provider.
Ultimately, his team CloudQuery in 'Data Lake' mode uses the “Ultra -Billy Infrastructure”, including AWS S3, ECS, Kleber, Athena and Lambda, to Venturebeat.
“We can quickly query every IP and discover who the owners are,” said Figueireedo. “We at the moment are capable of collect every part we have to be very low-cost with almost zero maintenance. This is the Holy Grail for our team.”
