Reminder and Agenda for Confidential Storage Spec Call - Feb 25 2021


Dmitri Zagidulin
 

Dear all,

This is a reminder that the DIF / CCG Secure Data Storage Working group weekly call will be happening on Thursday at 4pm Eastern / 1pm Pacific / 22:00 CEST.

Meeting link: https://us02web.zoom.us/j/84828031746?pwd=V0xGTnJ2Zm15RHlSRFpNTlRPQzdLUT09

Specification: https://identity.foundation/confidential-storage/
Specification repository: https://github.com/decentralized-identity/confidential-storage

Audio recordings and transcripts of previous meetings: https://dif.groups.io/g/sds-wg/wiki/19633

As always, the IPR policy requires that you can only make substantive contributions if you sign the IPR Release Form. Please follow the instructions at https://dif.groups.io/g/sds-wg/wiki/Home

Confidential Storage Spec Call Agenda
1. IPR Reminder
2. Introductions and Re-Introductions
3. Continuation of the ‘Division of Responsibilities between Hubs and EDVs’ discussion.
PLEASE REVIEW: Daniel Buchner's proposed list https://hackmd.io/qClYLUPkQ7uf0r3_4O7BUQ
4. Issue review

Thank you,
The Chairs


Manu Sporny
 

On 2/27/21 8:55 PM, Chris Were wrote:
I could break down the list we have discussed over the past few weeks and
point to the relevant section of CouchDB documentation that addresses those
requirements.
Yes, this is a next step. Ideally, we'd map every feature in the spec today to
CouchDB documentation. Chris, it would be good for you to volunteer to do this
since you seem to be the one most driven to demonstrate that this is true.
It's something we're going to have to document in time anyway, to demonstrate
why the work needs to be done (or that W3C shouldn't waste their time on it
because... CouchDB exists).

Once you document it (not just the list of requirements we've been going over
during the last month, but all the features in the existing spec as well), we
can review it as a group to see if there is consensus.

Zooming out a bit to look at the big picture... thinking about how SQL was
standardized may help. SQL was first standardized by ANSI in 1986... but there
were databases on the market for years that had the "relational database"
feature set... relational databases existed in 1970, 16 years before the
/first/ standard existed. SQL has been standardized ever since.. with the most
recent version being SQL:2019. That's 35 *years* of standardization! Why are
we still standardizing it!?

To replay your point back to you, but in a different context:

MariaDB (a popular open source relational database) covers many of the common
use case needs for relational databases... so why do we need an SQL standard?

With that in mind, some thoughts on the questions you asked:

This process feels like attempting to reinvent the wheel and produce a
sub-par outcome.
It always feels like that at first... like you're rubbing two sticks together
to create a fire when we already have nuclear power plants. Why don't we just
build a nuclear power plant? (Answer: there is no single specification on how
you build one and some of the skills are so specialized that it's out of the
reach of 99.99% of the population, which is not a good number if you're trying
to document and teach people how to do something).

The process of standards are not to innovate (at least, not primarily). It's
to look at everything that's out there and try to standardize the simplest set
of technologies that fit a Pareto distribution... that is, what 20% of
features meet 80% of the use cases. The goal isn't to get the standard to
support 100% of all use cases.

I understand CouchDB is not a specification, but as an implementation it's
pretty darn close to what we're looking for.
I have a vague concept of what your requirements are, Chris. :) I'm sure you
don't have a solid concept of what Digital Bazaar's requirements are...
or Transmute's... or SecureKey's... or Microsoft's... or Michael's. We're just
scratching the surface, these discussions take a LOOOONG time to get to a
basic understanding on everyone's *public* use cases.

... but the way we get there is to talk about it, and documentation of the
sort you're talking about is vital to that process.

Thoughts?

-- manu

--
Manu Sporny - https://www.linkedin.com/in/manusporny/
Founder/CEO - Digital Bazaar, Inc.
blog: Veres One Decentralized Identifier Blockchain Launches
https://tinyurl.com/veres-one-launches


Adrian Gropper
 

The lack of use-cases makes reaching consensus much harder since all we really have to build on is the charter. 

I spent some time today categorizing my personal data relationships and started a Use Cases document https://hackmd.io/q-uwOlisS0igaR2-vfXmlg

Please edit or add to the categories from your perspective and also add use cases. I would like to discuss the authorization perspective in a call.

If it is helpful to separate our work into EDVs and Hubs, then this thread https://lists.identity.foundation/g/sds-wg/message/82 also tries to address the authorization protocol(s).

- Adrian

On Sun, Feb 28, 2021 at 10:24 AM Manu Sporny <msporny@...> wrote:
On 2/27/21 8:55 PM, Chris Were wrote:
> I could break down the list we have discussed over the past few weeks and
> point to the relevant section of CouchDB documentation that addresses those
> requirements.

Yes, this is a next step. Ideally, we'd map every feature in the spec today to
CouchDB documentation. Chris, it would be good for you to volunteer to do this
since you seem to be the one most driven to demonstrate that this is true.
It's something we're going to have to document in time anyway, to demonstrate
why the work needs to be done (or that W3C shouldn't waste their time on it
because... CouchDB exists).

Once you document it (not just the list of requirements we've been going over
during the last month, but all the features in the existing spec as well), we
can review it as a group to see if there is consensus.

Zooming out a bit to look at the big picture... thinking about how SQL was
standardized may help. SQL was first standardized by ANSI in 1986... but there
were databases on the market for years that had the "relational database"
feature set... relational databases existed in 1970, 16 years before the
/first/ standard existed. SQL has been standardized ever since.. with the most
recent version being SQL:2019. That's 35 *years* of standardization! Why are
we still standardizing it!?

To replay your point back to you, but in a different context:

MariaDB (a popular open source relational database) covers many of the common
use case needs for relational databases... so why do we need an SQL standard?

With that in mind, some thoughts on the questions you asked:

> This process feels like attempting to reinvent the wheel and produce a
> sub-par outcome.

It always feels like that at first... like you're rubbing two sticks together
to create a fire when we already have nuclear power plants. Why don't we just
build a nuclear power plant? (Answer: there is no single specification on how
you build one and some of the skills are so specialized that it's out of the
reach of 99.99% of the population, which is not a good number if you're trying
to document and teach people how to do something).

The process of standards are not to innovate (at least, not primarily). It's
to look at everything that's out there and try to standardize the simplest set
of technologies that fit a Pareto distribution... that is, what 20% of
features meet 80% of the use cases. The goal isn't to get the standard to
support 100% of all use cases.

> I understand CouchDB is not a specification, but as an implementation it's
> pretty darn close to what we're looking for.

I have a vague concept of what your requirements are, Chris. :) I'm sure you
don't have a solid concept of what Digital Bazaar's requirements are...
or Transmute's... or SecureKey's... or Microsoft's... or Michael's. We're just
scratching the surface, these discussions take a LOOOONG time to get to a
basic understanding on everyone's *public* use cases.

... but the way we get there is to talk about it, and documentation of the
sort you're talking about is vital to that process.

Thoughts?

-- manu

--
Manu Sporny - https://www.linkedin.com/in/manusporny/
Founder/CEO - Digital Bazaar, Inc.
blog: Veres One Decentralized Identifier Blockchain Launches
https://tinyurl.com/veres-one-launches







Chris
 

Hi All,

The recent conversation discussing Hub / EDV spec has been extremely useful as it has helped shape the key areas the spec needs to consider and draw some clearer boundaries around requirements.

That being said, it has reinforced for me how the majority of the items being discussed are already well implemented and battle tested within CouchDB (which already runs across multiple platforms and protocols). This process feels like attempting to reinvent the wheel and produce a sub-par outcome. I could break down the list we have discussed over the past few weeks and point to the relevant section of CouchDB documentation that addresses those requirements. I understand CouchDB is not a specification, but as an implementation it's pretty darn close to what we're looking for. It seems to make sense to have a meaningful conversation about how that gap could be bridged to speed up this process and result in a much richer feature set?

I probably sound like a broken record as I've brought this up multiple times, but have not received any meaningful response as to why CouchDB, it's learnings and it's advanced feature set are being ignored.

There's currently a tendency to put stuff out of scope because it's too hard / complex to work within the current implementations compliant with the spec. That feels a bit like the tail wagging the dog. Is that the real criteria we should be using? That's definitely a consideration, but more important is what are the essential capabilities required for a useful v1 spec.

On that front, we also need to consider performance. If we end up with a spec that requires polling and has restricted replication capabilities, it's going to limit its use for many real world use cases and implementers will seriously consider using non-compliant solutions.

As it stands, the spec as it's currently shaping risks having limited utility and will require many non-compliant "add ons" to make it usable for many implementers. With v1 of the spec being a long way off I'm not sure we're positioning ourselves for long term success. At the very least, I agree with Michael's suggestion of breaking the spec up and modularising it's capabilities to provide greater flexibility for implementers in the future.

Cheers,
Chris

On Fri, 26 Feb 2021 at 06:05, Dmitri Zagidulin <dzagidulin@...> wrote:
Dear all,

This is a reminder that the DIF / CCG Secure Data Storage Working group weekly call will be happening on Thursday at 4pm Eastern / 1pm Pacific / 22:00 CEST.

Meeting link: https://us02web.zoom.us/j/84828031746?pwd=V0xGTnJ2Zm15RHlSRFpNTlRPQzdLUT09

Specification: https://identity.foundation/confidential-storage/
Specification repository: https://github.com/decentralized-identity/confidential-storage

Audio recordings and transcripts of previous meetings: https://dif.groups.io/g/sds-wg/wiki/19633

As always, the IPR policy requires that you can only make substantive contributions if you sign the IPR Release Form. Please follow the instructions at https://dif.groups.io/g/sds-wg/wiki/Home

Confidential Storage Spec Call Agenda
1. IPR Reminder
2. Introductions and Re-Introductions
3. Continuation of the ‘Division of Responsibilities between Hubs and EDVs’ discussion.
PLEASE REVIEW: Daniel Buchner's proposed list https://hackmd.io/qClYLUPkQ7uf0r3_4O7BUQ
4. Issue review

Thank you,
The Chairs


Chris
 

Thanks Manu for the thoughtful reply.
 
Yes, this is a next step. Ideally, we'd map every feature in the spec today to
CouchDB documentation. Chris, it would be good for you to volunteer to do this
since you seem to be the one most driven to demonstrate that this is true.
It's something we're going to have to document in time anyway, to demonstrate
why the work needs to be done (or that W3C shouldn't waste their time on it
because... CouchDB exists). 
Once you document it (not just the list of requirements we've been going over
during the last month, but all the features in the existing spec as well), we
can review it as a group to see if there is consensus.

Sure, I'm happy to make a first pass on this. Any thoughts on the best format / structure?
 
Zooming out a bit to look at the big picture... thinking about how SQL was
standardized may help. ... <snip>

Totally agree, standardizing SQL is a perfect example.
 
To replay your point back to you, but in a different context:

MariaDB (a popular open source relational database) covers many of the common
use case needs for relational databases... so why do we need an SQL standard?

To clarify, I'm not arguing we don't need a standard. To work with the same analogy -- if we're building an SQL standard we should be looking very closely at MariaDB (and other prior art / SQL-like implementations) when developing that standard.
 
The process of standards are not to innovate (at least, not primarily). It's
to look at everything that's out there and try to standardize the simplest set
of technologies that fit a Pareto distribution... that is, what 20% of
features meet 80% of the use cases. The goal isn't to get the standard to
support 100% of all use cases.

Yes, I agree with that.

I guess one of my concerns relates to feeling like we're looking closely enough at what's out there, but as noted above I'm happy to make a start on that with CouchDB. Michael has mentioned some other prior-art (in the replication context) that could be reviewed here. I'm sure there's more and across other contexts? Is there other work to document all the prior art that I have missed?

The point of meeting 80% of use cases is important, but there's some subtlety in that. Is it 80% of the "total use cases" or 80% of the "most important use cases" -- those are two very different things. And are we dropping use cases because they're "too hard with the current spec implementation" or because "they're not that important"?

Regardless, the ability to have a framework of modularising or extending the capabilities seems like something we should seriously consider. That will minimise the risk of developers ignoring the spec because they're locked in to only working with what's provided out of the box.
 
> I understand CouchDB is not a specification, but as an implementation it's
> pretty darn close to what we're looking for.

I have a vague concept of what your requirements are, Chris. :) I'm sure you
don't have a solid concept of what Digital Bazaar's requirements are...
or Transmute's... or SecureKey's... or Microsoft's... or Michael's. We're just
scratching the surface, these discussions take a LOOOONG time to get to a
basic understanding on everyone's *public* use cases.

... but the way we get there is to talk about it, and documentation of the
sort you're talking about is vital to that process.

Totally agree :) 


Michael Herman (Trusted Digital Web)
 

RE: Michael has mentioned some other prior-art (in the replication context) that could be reviewed here. I'm sure there's more and across other contexts? Is there other work to document all the prior art that I have missed?

 

A few weeks ago I highlighted that I was working on a whitepaper to document and diagram the architectural variations for the 4 key layers in a confidential/trust content storage solution. I hope to have a first complete draft “later this week”.  You can find a link to a copy of today’s snapshot at the bottom of this email.

 

The Preface explains the twists and turns this document has taken and hence, why it has the title that it has…

Context

 

“Sometimes called “reasoning from first principles,” the idea is to break down complicated problems into basic elements and then reassemble them from the ground up. It's one of the best ways to learn to think for yourself, unlock your creative potential, and move from linear to non-linear results.”

[First Principles: The Building Blocks of True Knowledge(https://fs.blog/2018/04/first-principles/)]

“I think it is most important to reason from first principles rather than by analogy. One of the ways we conduct our lives is we reason by analogy. We do this because something was like something else that was done or it was like what other people were doing. It’s mentally easier to reason by analogy rather than from first principles.”

[First Principles Method Explained by Elon Musk (https://www.youtube.com/watch?v=NV3sBlRgzTI)]

 

Preface

This is a first-principles software architecture reference model (ARM) whitepaper that documents the comprehensive analysis of the potential architecture variations for the Trust Content Storage Stack (TCS Stack) including detailed interim as well as final assessments and recommendations.

 

The intent and purpose of this document has taken several strategic turns during the course of its development. Originally entitled Secure Data Storage Working Group (sds-wg) Confidential Storage (CS): Functional Architecture Reference Models (CS-FARMs), the initial goal was to help evolve the SDS Layers diagram in the Confidential Storage 1.0 Specification (https://identity.foundation/confidential-storage/#ecosystem-overview) to make it more useful for discussing the technology and architectural options for supported replication between encrypted data vaults.

 

This whitepaper evolved quickly over the subsequent weeks from being a few pages in length to more than 200 pages. The whitepaper had forked from its original mission. The purpose of the current version of the whitepaper to fully describe the architecture variations for a new project:  Trusted Content Storage Architecture (TCS Stack).

 

The work documented here was performed under the auspices of the Trusted Digital Web project in the Hyperonomy Digital Identity Lab of Parallelspace Corporation.

 

NOTE: The publication of this document coincides with recent discussions (January-February 2021) about secure data storage solutions in the Decentralized Identity Foundation (DIF) Secure Data Storage working group (sds-wg) that were taking place during the development of the Confidential Storage specification. This is not a DIF publication, unofficial, official, or otherwise.

 

Here’s a commenter link to a copy of today’s snapshot: https://drive.google.com/file/d/1-Ve263KT7svN7oppIHyhK6idXikcbBsn/view?usp=sharing

 

Any and all feedback is greatly appreciated.

 

Best regards,

Michael Herman

Sovrin Foundation Self-Sovereignist

 

Self-Sovereign Blockchain Architect

Trusted Digital Web

Hyperonomy Digital Identity Lab

Parallelspace Corporation

 

 

 

 

From: sds-wg@... <sds-wg@...> On Behalf Of Chris
Sent: March 2, 2021 8:38 PM
To: Manu Sporny <msporny@...>
Cc: sds-wg@...; sds-wg@dif.groups.io; Credentials Community Group <public-credentials@...>
Subject: Re: [sds-wg] Reminder and Agenda for Confidential Storage Spec Call - Feb 25 2021

 

Thanks Manu for the thoughtful reply.

 

Yes, this is a next step. Ideally, we'd map every feature in the spec today to
CouchDB documentation. Chris, it would be good for you to volunteer to do this
since you seem to be the one most driven to demonstrate that this is true.
It's something we're going to have to document in time anyway, to demonstrate
why the work needs to be done (or that W3C shouldn't waste their time on it
because... CouchDB exists). 

Once you document it (not just the list of requirements we've been going over
during the last month, but all the features in the existing spec as well), we
can review it as a group to see if there is consensus.

 

Sure, I'm happy to make a first pass on this. Any thoughts on the best format / structure?

 

Zooming out a bit to look at the big picture... thinking about how SQL was
standardized may help. ... <snip>

 

Totally agree, standardizing SQL is a perfect example.

 

To replay your point back to you, but in a different context:

MariaDB (a popular open source relational database) covers many of the common
use case needs for relational databases... so why do we need an SQL standard?

 

To clarify, I'm not arguing we don't need a standard. To work with the same analogy -- if we're building an SQL standard we should be looking very closely at MariaDB (and other prior art / SQL-like implementations) when developing that standard.

 

The process of standards are not to innovate (at least, not primarily). It's
to look at everything that's out there and try to standardize the simplest set
of technologies that fit a Pareto distribution... that is, what 20% of
features meet 80% of the use cases. The goal isn't to get the standard to
support 100% of all use cases.

 

Yes, I agree with that.

 

I guess one of my concerns relates to feeling like we're looking closely enough at what's out there, but as noted above I'm happy to make a start on that with CouchDB. Michael has mentioned some other prior-art (in the replication context) that could be reviewed here. I'm sure there's more and across other contexts? Is there other work to document all the prior art that I have missed?

 

The point of meeting 80% of use cases is important, but there's some subtlety in that. Is it 80% of the "total use cases" or 80% of the "most important use cases" -- those are two very different things. And are we dropping use cases because they're "too hard with the current spec implementation" or because "they're not that important"?

 

Regardless, the ability to have a framework of modularising or extending the capabilities seems like something we should seriously consider. That will minimise the risk of developers ignoring the spec because they're locked in to only working with what's provided out of the box.

 

> I understand CouchDB is not a specification, but as an implementation it's
> pretty darn close to what we're looking for.

I have a vague concept of what your requirements are, Chris. :) I'm sure you
don't have a solid concept of what Digital Bazaar's requirements are...
or Transmute's... or SecureKey's... or Microsoft's... or Michael's. We're just
scratching the surface, these discussions take a LOOOONG time to get to a
basic understanding on everyone's *public* use cases.

... but the way we get there is to talk about it, and documentation of the
sort you're talking about is vital to that process.

 

Totally agree :) 


Manu Sporny
 

On 3/2/21 10:37 PM, Chris Were wrote:
Sure, I'm happy to make a first pass on this. Any thoughts on the best
format / structure?
Great! Thanks Chris.

I suggest Google Docs to start -- that'll help everyone collaborate using
tooling most everyone is used to and provide feedback.

The structure is up to you, of course. I would suggest that you try listing
all the features of EDVs and how CouchDB maps to those features. You might
also want to take the opportunity to highlight the features that you feel are
important.

To clarify, I'm not arguing we don't need a standard. To work with the same
analogy -- if we're building an SQL standard we should be looking very
closely at MariaDB (and other prior art / SQL-like implementations) when
developing that standard.
Yes, agreed. :)

The point of meeting 80% of use cases is important, but there's some
subtlety in that. Is it 80% of the "total use cases" or 80% of the "most
important use cases" -- those are two very different things. And are we
dropping use cases because they're "too hard with the current spec
implementation" or because "they're not that important"?
That's an excellent question. I expect it's mostly the latter, but it may be
coming across as the former.

Typically, when groups can't agree, they inject optionality into the
specification... which exchanges the short term problem of agreeing on one
solution with the longer term problem of interop challenges. That said,
extensible specifications tend to last longer than rigid ones. It's a
trade-off, like many other engineering decisions. :)

In any case, sounds like there is alignment on the next step (and the general
philosophy), which is good. :)

-- manu

--
Manu Sporny - https://www.linkedin.com/in/manusporny/
Founder/CEO - Digital Bazaar, Inc.
blog: Veres One Decentralized Identifier Blockchain Launches
https://tinyurl.com/veres-one-launches


Michael Herman (Trusted Digital Web)
 

Here’s a better link (I think): https://docs.google.com/document/d/1-Ve263KT7svN7oppIHyhK6idXikcbBsn/edit

 

From: Michael Herman (Trusted Digital Web) <mwherman@...>
Sent: March 3, 2021 1:46 AM
To: sds-wg@...; Manu Sporny <msporny@...>; Chris <chris@...>
Cc: sds-wg@dif.groups.io; Credentials Community Group <public-credentials@...>
Subject: RE: [sds-wg] Reminder and Agenda for Confidential Storage Spec Call - Feb 25 2021

 

RE: Michael has mentioned some other prior-art (in the replication context) that could be reviewed here. I'm sure there's more and across other contexts? Is there other work to document all the prior art that I have missed?

 

A few weeks ago I highlighted that I was working on a whitepaper to document and diagram the architectural variations for the 4 key layers in a confidential/trust content storage solution. I hope to have a first complete draft “later this week”.  You can find a link to a copy of today’s snapshot at the bottom of this email.

 

The Preface explains the twists and turns this document has taken and hence, why it has the title that it has…

Context

 

“Sometimes called “reasoning from first principles,” the idea is to break down complicated problems into basic elements and then reassemble them from the ground up. It's one of the best ways to learn to think for yourself, unlock your creative potential, and move from linear to non-linear results.”

[First Principles: The Building Blocks of True Knowledge(https://fs.blog/2018/04/first-principles/)]

“I think it is most important to reason from first principles rather than by analogy. One of the ways we conduct our lives is we reason by analogy. We do this because something was like something else that was done or it was like what other people were doing. It’s mentally easier to reason by analogy rather than from first principles.”

[First Principles Method Explained by Elon Musk (https://www.youtube.com/watch?v=NV3sBlRgzTI)]

 

Preface

This is a first-principles software architecture reference model (ARM) whitepaper that documents the comprehensive analysis of the potential architecture variations for the Trust Content Storage Stack (TCS Stack) including detailed interim as well as final assessments and recommendations.

 

The intent and purpose of this document has taken several strategic turns during the course of its development. Originally entitled Secure Data Storage Working Group (sds-wg) Confidential Storage (CS): Functional Architecture Reference Models (CS-FARMs), the initial goal was to help evolve the SDS Layers diagram in the Confidential Storage 1.0 Specification (https://identity.foundation/confidential-storage/#ecosystem-overview) to make it more useful for discussing the technology and architectural options for supported replication between encrypted data vaults.

 

This whitepaper evolved quickly over the subsequent weeks from being a few pages in length to more than 200 pages. The whitepaper had forked from its original mission. The purpose of the current version of the whitepaper to fully describe the architecture variations for a new project:  Trusted Content Storage Architecture (TCS Stack).

 

The work documented here was performed under the auspices of the Trusted Digital Web project in the Hyperonomy Digital Identity Lab of Parallelspace Corporation.

 

NOTE: The publication of this document coincides with recent discussions (January-February 2021) about secure data storage solutions in the Decentralized Identity Foundation (DIF) Secure Data Storage working group (sds-wg) that were taking place during the development of the Confidential Storage specification. This is not a DIF publication, unofficial, official, or otherwise.

 

Here’s a commenter link to a copy of today’s snapshot: https://drive.google.com/file/d/1-Ve263KT7svN7oppIHyhK6idXikcbBsn/view?usp=sharing

 

Any and all feedback is greatly appreciated.

 

Best regards,

Michael Herman

Sovrin Foundation Self-Sovereignist

 

Self-Sovereign Blockchain Architect

Trusted Digital Web

Hyperonomy Digital Identity Lab

Parallelspace Corporation

 

 

 

 

From: sds-wg@... <sds-wg@...> On Behalf Of Chris
Sent: March 2, 2021 8:38 PM
To: Manu Sporny <msporny@...>
Cc: sds-wg@...; sds-wg@dif.groups.io; Credentials Community Group <public-credentials@...>
Subject: Re: [sds-wg] Reminder and Agenda for Confidential Storage Spec Call - Feb 25 2021

 

Thanks Manu for the thoughtful reply.

 

Yes, this is a next step. Ideally, we'd map every feature in the spec today to
CouchDB documentation. Chris, it would be good for you to volunteer to do this
since you seem to be the one most driven to demonstrate that this is true.
It's something we're going to have to document in time anyway, to demonstrate
why the work needs to be done (or that W3C shouldn't waste their time on it
because... CouchDB exists). 

Once you document it (not just the list of requirements we've been going over
during the last month, but all the features in the existing spec as well), we
can review it as a group to see if there is consensus.

 

Sure, I'm happy to make a first pass on this. Any thoughts on the best format / structure?

 

Zooming out a bit to look at the big picture... thinking about how SQL was
standardized may help. ... <snip>

 

Totally agree, standardizing SQL is a perfect example.

 

To replay your point back to you, but in a different context:

MariaDB (a popular open source relational database) covers many of the common
use case needs for relational databases... so why do we need an SQL standard?

 

To clarify, I'm not arguing we don't need a standard. To work with the same analogy -- if we're building an SQL standard we should be looking very closely at MariaDB (and other prior art / SQL-like implementations) when developing that standard.

 

The process of standards are not to innovate (at least, not primarily). It's
to look at everything that's out there and try to standardize the simplest set
of technologies that fit a Pareto distribution... that is, what 20% of
features meet 80% of the use cases. The goal isn't to get the standard to
support 100% of all use cases.

 

Yes, I agree with that.

 

I guess one of my concerns relates to feeling like we're looking closely enough at what's out there, but as noted above I'm happy to make a start on that with CouchDB. Michael has mentioned some other prior-art (in the replication context) that could be reviewed here. I'm sure there's more and across other contexts? Is there other work to document all the prior art that I have missed?

 

The point of meeting 80% of use cases is important, but there's some subtlety in that. Is it 80% of the "total use cases" or 80% of the "most important use cases" -- those are two very different things. And are we dropping use cases because they're "too hard with the current spec implementation" or because "they're not that important"?

 

Regardless, the ability to have a framework of modularising or extending the capabilities seems like something we should seriously consider. That will minimise the risk of developers ignoring the spec because they're locked in to only working with what's provided out of the box.

 

> I understand CouchDB is not a specification, but as an implementation it's
> pretty darn close to what we're looking for.

I have a vague concept of what your requirements are, Chris. :) I'm sure you
don't have a solid concept of what Digital Bazaar's requirements are...
or Transmute's... or SecureKey's... or Microsoft's... or Michael's. We're just
scratching the surface, these discussions take a LOOOONG time to get to a
basic understanding on everyone's *public* use cases.

... but the way we get there is to talk about it, and documentation of the
sort you're talking about is vital to that process.

 

Totally agree :)