Software Defined Storage
Ceph RGW Message Queue API for Serverless Computing
February 6, 2021
3:10 PM – 4:10 PM Add to calendar
3:10 PM – 4:10 PM Add to calendar
D.sds
A proposal to support AWS SQS API natively in Ceph RGW, for advanced Serverless computing use cases.
Bucket notification has become widespread in many applications including AI/ML and Edge Computing. We have demonstrated that Ceph RGW bucket notifications trigger Serverless functions including Knative and AWS Lambda. Such workflows can be characterized as event push and event driven.
We’ve identified yet another workflow that allows Serverless computing frameworks to be more preemptive in auto scaling. The workflow is based on the event queue polling model, the opposite of event push, that allows Serverless computing frameworks, such as KEDA, to preemptively scale the functions based on the queue length. Such workflow is more advantageous and lightweighted in dealing with highly bursty event traffic and reducing Serverless function cold start overhead.
In order to support this workflow, we propose a new set of message queue APIs, modeled after AWS SQS, for Ceph RGW. In this talk, we’ll present the overview and planning of this technology to the community.
Additional information
| Type | devroom |
|---|
More sessions
| 2/6/21 |
There are a number of available tools and approaches for benchmarking S3 performance in Ceph. Benchmarking reliably and sensibly gets more challenging as you scale out storage clusters, gateways, and benchmarking drivers. In this talk I’ll go over some of the benchmarking tools we’ve used in recent months, talk about our focus around COSBench, and some of the approaches and steps we took to improve the state of COSBench for Ceph benchmarking, both in COSBench itself and with helper tools.
|
| 2/6/21 |
What's new in Ceph RGW Ceph is an open source, highly scalable, software defined storage that provides object, file and block interfaces under a unified system. Ceph Object Storage Gateway (RGW) provides a RESTful object storage interface to the Ceph Storage cluster. It provides an interface that is compatible with a large subset of AWS S3 APIs. In this talk we will discuss the upcoming features in ‘Pacific’ release and beyond. The list of features are: S3select Lua scripting Encryption - ...
|
| 2/6/21 |
Data Processing Units (DPUs) and SmartNICs--these things are getting a lot of attention, particularly in the network storage world. There's a bit of history behind them, particularly if you consider TCP Offload Engine (TOE) cards and iSCSI NICs, both of which have been around for quite a while. This latest wave, however, is more general-purpose by design and presents an opportunity for developers like us.
|
| 2/6/21 |
LINSTOR enables the automation of the deployment of block storage that is replicated over multiple storage nodes and can act as a software defined storage provider for various container and virtualization platforms, such as Kubernetes, OpenStack, OpenNebula and Proxmox. It also includes additional storage features, such as the automation of storage encryption, deduplication or the automatic creation of file systems on new block storage volumes.
|
| 2/6/21 |
Kubernetes graduated Container Storage Interface (CSI) to GA status in v1.13. Since CSI’s origin, it has evolved to support a large number of vendors and storage formats. It has brought the industry together in consensus about the best practices in storage. However, CSI’s major limitation is that it only supports Block and File storage. The rapid growth of Object storage means that a growing share of the ecosystem is currently using ad-hoc solutions to leverage this new technology. Object ...
|
