Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

VQL

No description
by

Aaron Bawcom

on 13 February 2016

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of VQL

VQL
Data
Description

Data Query
Grammar

Data
Federation

Data
Processing

{
objects : [ { sysid : [ dallas, 0, adapter, 7413 ], props : [ { name : "foo" }, { status : "running" } ] }, ... ],
edges : [ [ sysid_a, sysid_z ], ... ],
symbols : [ { symbol : 2 }, ...]
}
Document Oriented
Above is a JSON'y Example
VQL objects/edges can be represented
either in JSON or ETF
sysid: [ Geo-Location, Instance-Location, ObjectType, Instance ]
Definition
Data
Storage

Data
Modeling

Data
Virtualization

Virtual
Query
Language

volume.isolation = true and volume.disk_type = sata project vm(name)

tenant.name = A project vserver | project pswitch in tenant.name = A project vm | project pswitch
Tenant Model
Parallel DQP Set
Intersection
pSwitch Model
vServer Model
VM Model
Sequential
DQP Pipeline
Optional
Transmission
Footprint
Optimization
vServer, Volume, Controller, Aggregate, Cluster, Disk
Links
vServer-Volume
Controller-vServer
Cluster-Controller
Aggregate-Volume
Aggreaget-Disk
Model Provider 1
VM, Host, Datastore
VMDatastoreMount
Links
VM-Datastore
VM-Host
Host-Datastore
Datastore-Volume
VMDatastoreMount-VM
VMDatastoreMount-Datastore
Model Provider 2
MetricValue
Links
VM-MV
Host-MV
Datastore-MV
VMDatastore-MV
Model Provider 3
Models
Setters
vServer.*
Cluster.*
Aggregate.*
Volume.*
Controller.Prop1
Controller.Prop2
Models
Setters
...
Models
Setters
...

tenant.name = alpha project vserver | project pswitch in tenant.name = A project vm | project pswitch
Model Providers offer individual data models as well as link handling to other data types

tenant.name = alpha project storage_cluster | project storage_policy | set storage_policy.type = Gold
Setters provide the ability to call procedural logic declaratively
Data
Applications

Supported Models
vm, host, datastore
Links
vm-host
host-datastore
Setters
vm.*
host.prop1
host.prop2
datastore.prop1
datastore.prop2
Model Provider
Supported Models
port, vlan, pswitch, channel, route-table, route
Links
pswitch-port
vlan-port
vlan-route
...
Setters
vlan.*
port.duplex
channel.type
Model Provider
Supported Models
metric_value
Links
mv-*
Setters
mv.value
mv.timeframe
Model Provider
App Profile
Supported Models
volume, vserver, dotcluster
controller, aggregate, disk
Links
vol-aggregate
aggregate-disk
...
Setters
vserver.*
dotcluster.*
volume.*
Model Provider
hosts
datastores
dotcluster
controllers
vservers
pswitches
ports
route-table
routes
vlans
Supported Models
vm, host, datastore
Links
vm-host
host-datastore
Setters
vm.*
host.prop1
host.prop2
datastore.prop1
datastore.prop2
Model Provider (VMware)
Supported Models
vm, host, datastore
Links
vm-host
host-datastore
Setters
vm.*
host.prop1
host.prop2
datastore.prop1
datastore.prop2
Model Provider (Hyper-V)
Supported Models
vm, host, datastore
Links
vm-host
host-datastore
Setters
vm.*
host.prop1
host.prop2
datastore.prop1
datastore.prop2
Model Provider (KVM)
vm.cpu > 2 project volume
VQL Federated Query Procesor
{
operation: set,
data : {
objects : [ { sysid : [ dallas, 0, adapter, 7413 ], props : [ { name : "foo" }, { status : "running" } ] }, ... ],
edges : [ [ sysid_a, sysid_z ], ... ],
symbols : [ { symbol : 2 }, ...]
}
}
Graph based storage
All I/O via Application Network - RabbitMQ federation
Set & delete operations
Collection per object type - MongoDB
Geographically diverse availability
Ensures global uniqueness for all objects via sysid service
Named Edge Types
Per Property Historical
Store
Instruction
Database
Persistence
Real-Time
Processors
Applications
1. Store
2. Persist
3. Acknowledgement
4. Send to subscribing
Real-Time Queries
5. Send to subscribing
Applications
Create new objects, properties, relationships based on sophisticated procedural logic
Example - Find bottlenecks in the network between an application and a storage target
Create a network_path object that consists of edges with a VM, a volume/vserver/etc. and all physical network ports between those two endpoints. The application would rely on existing virtualization (VM), storage(volume), and networking(pport) data providers to create new data objects that provide higher level correlation between all of the existing objects.
Distributed Real-Time VQL Query Processing
Ad-Hoc VQL query results
Real-Time differential updates
Object compression
Compute-Local applications
Batch streaming of Ad-Hoc or differential results
Processor
Processor
Processor
Processor
Query1: QueryOutput( query2 ) | mv.key='vdisk.latency' project hrollup(vm)
Query2: mv.key='vdisk.latency' and mv.timeunit='1min' project trollup(timeunit=1hour)
Query1
Query2
query results
streamed in
real-time
Query Director
Load balances queries
across processing nodes
Real-Time
Data Input
Real-Time
Data Output
Actor Model Based
Complex Event Processing
Pure Message Passing
vm.cpu > 2 and vm.mem_gb > 8192 project host
vm.cpu>2
vm.mem>8192
and
project host
vm.cpu properties
vm.mem properties
only vm sysid's
who have a cpu
property
value > 2
only vm sysid's
who have a mem
property
value > 8192
vm-host edges
vm sysid's
that were met
previous criteria
only host objects
who are connected to
vm objects who have
a cpu value > 2 and
a mem value > 8192
Graph based Key-Value Grammar
Declarative Statements of data structure
Support for standard Set semantics
Shortest path edge traversal
Used for declarative policy encoding
Partial object support
Returns all vm names that are connected to any volume
that provides isolation and uses only SATA disk types
Returns all pswitch objects that are connected to tenant A
either through virtual machines or vservers connected to
tenant A
Store
Instruction
Data
Persistence
Real-Time
Processors
Optimize
Application
1. VM I/O Data
3/12. Persist
4/13. Acknowledgement
5/14. Send to subscribing
Real-Time Queries
7. Send new policy match to
subscribing application
Automated SLO Enforcement
Setters allow for the declarative nature of VQL to back-end into procedural logic to interface with specific API's provided by element management systems
Setters can be atomic such as cpu = 2
Setters can be transactional such as cpu = { cores : 2, speed: 3Ghz }
Setters can interface with any type of system including SOAP, REST, DB, Rabbit messages, files, etc.
Harvester
(Virt)
Element Manager
(Virt)
Setter
Handler
Element Manager
(Storage)
Harvester
(Storage)
2. Store
6. The new state feeds into
separate policy based criteria
8. Call appropriate
Setter(s)
9. Call specific Element
Manager API method(s)
10. Receive new state
15. Send new
policy match
to subscribing
application
11. Store
15.Acknowledge
Successful
Settter
This will ultimately become part
of the Real-Time Processor as well
Example
Optimize storage based on observed behavior within the virtualized environment
Babel Messaging
Framework

Data Geography
Chicago
New York
Dallas
Chicago
Shard
Cluster
Dallas
Shard
Cluster
Reflex Node
DQP
MongoS: Chicago
MongoS: Dallas
MongoS: New York
MongoS: Dallas
MongoS: Chicago
MongoS: New York
Dallas
Replica
Set
New York
Replica
Set
sysid
location
Reflex Node
DQP
sysid
location
Chicago
Replica
Set
New York
Replica
Set
New York
Shard
Cluster
MongoS: New York
MongoS: Chicago
MongoS: Dallas
Reflex Node
DQP
sysid
location
Chicago
Replica
Set
Dallas
Replica
Set
DQP
Traffic

Persistence
Replication

Managing at scale requires support for geographic redundancy and load distribution for both intra and inter geographic locations.
Dallas
Non
Replicated
Data
Policies are just VQL queries. The VQL query acts as
a sophisticated language for specifying interdependent criteria
Monitor
Analyze
Plan
Execute
Knowledge
A service abstraction on top of AMQP for messaging
between a large network of services that are potentially
using different languages. Babel makes it very simple to build distributed services. Babel provides semantics for:
Single-Call, Single-Response
Single-Call, Multi-Response
Multi-Call, Single-Response
Multi-Call, N-Responses
Multi-Call, Multi-Response
Large Binary Send
Large Binary Receive
Async/Sync
1 way streams
2 way streams
HA/Load Balancing across service instances
mv.type=vdisk and mv.key = 'read.latency.ms' and mv.value < 150
Example: SLO ensuring vdisk performance levels
Legend
Supported Models
disk, aggregate, volume, vserver
controller, qtree
Links
voulme-vm
vserver-controller
aggregate-disk
Setters
vserver.*
controller.prop1
controller.prop2
volume.prop1
volume.prop2
Model Provider (NetApp)
Chicago
Non
Replicated
Data
New York
Non
Replicated
Data
Deployment
Packaging

Component Repository
Master Manifest
Code Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
Component Repository
New
Install
Combined
Upgrade
Customer Environment
Build
Process
Individual
Component
Builds
Genesis
Node
APP 2
APP 1
APP 4
APP 6
APP 1
APP 7
APP 7
APP 6
APP 3
APP 5
Software
Repository
Application
APP 2
APP 3
Current Language Support
Java
C#
Python
JavaScript
Erlang

AMQP Based
Application Network

Deployment Topologies
Multiple application topologies are possible to use for different application needs across a single fully connected graph. A RabbitMQ broker exists on each node. If proper underlying network connectivity exists, when a new node is deployed from a factory/repository node it automatically connects to the appropriate part of the application network and establishes secure RabbitMQ federation.
Mesh
Application
Hierarchical Application
Hub-Spoke
Application
HA
Services
dmsd
dsd
reposervice
heartbeat
authd
harvestmgrd
jobservice
topology
queryd
napsd
deployservice
mvs

amqp_client
apkg
authd
babeljs
babeljs
blackbox
blush
blush-admin
blush-capacity
blush-client-common
blush-dashboard
blush-deploy
blush-inventory
blush-services
blush-tenant
blush-vql-designer
blush-workflow
busey
ceres
ceres_app
clio
cloudos
connection_broker
core
couchdb
data_service
data_service_app
defaults
dingo
dms
dms_app
dms_client
dqp
drone
ds_client
edgar
eep
emongo
erlang
erlang-rfxlib
erlang-strapper
eseries
eseries_app
harvest_mgr
hyperion
hyperion_app
ivy
java
jobsrv
jobsrv_app
kernel
lager
lichen
link_builder_cache
litmus
metastasizer
model_builder_cache
monarch
mongodb
mono
naerl
node-bertrpc
nolo
nolo_app
npm-registry
os
ovftool
python
python-babel
python-misc
python-rfxlib
qs_client
query_registrar
query_registrar_cache
rabbit-ca
reflex_topic_notifier
repo_service
rulio
sesh
setter_network_client
setuphelper

sshview
stratus
streaming_client
tako-fdpass
unagi
vep
vierl
vol
vqlapi
vsphere
winerl
Applications
Remote data access - API into external system
Local data access - Local data storage
Data Proximity
Model A
Windows
Host
API 1
API 2
Management
System
Type 1
Model B
SC
Node
API 1
API 2
Management
System
Type 2
Model C
API 1
API 2
Storage
Controller Type 1
Hypervisor
Type 1
Switch
Type 1
Model M
SC
Collector
Node
Model N
SC
Data Storage
Infrequent
(data doesn't change frequently)
SC
Data Storage
Frequent
(data changes frequently)
Model X
Distributed Query Processor
X.prop1 > 50 project A | A.val = 1 project B without N.prop2 < 65 project M | m.prop3 = foo project X | X.prop1 > 0 project B | B.prop4 = bar
Babel Dataservice
Provided within
Windows Host
External Management Systems
Model Providers map VQL to API calls
Element Systems that provide source data to
collectors that store data in VQL Data model
1
2
3
This method has not yet been proven in a lab environment but is based on the proven existence of all building blocks that enable such as RabbitMQ being able to run on Windows
Disclaimer
Full transcript