Continuations: System and Organizational Scaling - Continuations
tweetip
· 1 year ago
We're big fans of API's - unfortunately, they do break. Each rev of twitter has had an api glitch - often an issue not easily detected.
As important as scaling, are methods for auditing the feed from the api as it flows and joins with other feeds.
A comment on AVC mentions the Financial industry figured this out long ago. We agree - but their budget is in the 100's of millions - each year :)
albert
· 1 year ago
Agree that APIs need a good approach to versioning for changes that are not backward compatible. Ideally though, most changes to an API make it more powerful without breaking existing apps.
tweetip
· 1 year ago
Hi Albert. Most often, our point of view takes awhile to manifest:
Twitter API Fail ~ Location
Foremost, our point of view comes from twitter stating they are becoming a “global communication utility”.
... more on our tumblelog, or on twitter dev blog, or via shortened url :) http://tweetip.us/lkrha
albert
· 1 year ago
I read the post on your tumblelog, but am not sure I understand what the failure in the Twitter API is.
tweetip
· 1 year ago
from a mission critical developer/end user point of view, and twitter being a ---> worldwide comm utility <---
by not drafting an advanced specification, twitter failed to insure the integrity of the data. Mobile app developers just started pumping in whatever format/values they desired. This data "breaks" our app, if we are filtering values based on historical analysis as is the case of First Responder / Earthquake / Tornado / Emergency Services or for that matter Market Movement / Breaking News / any realtime anything where we feel Location is integral to our analysis.
There's more on this convo at the twitter google dev blog.
albert
· 1 year ago
Thanks for clarifying -- I understand this better now.
tweetip
· 1 year ago
Respectively, we submit the beginnings of the firestorm we've politely tried to make you aware of:
twitter just broke API 4 tons of apps by introducing rate limits 2 unauth calls - way 2 kill app ecology —via @emp
albert
· 1 year ago
Ouch - that's not good that they broke stuff but I understand why they would want rate limits.
mccv
· 1 year ago
I think the challenge here is that eventually somebody will want to add a horizontal layer that encompasses multiple vertical slices. Usually this is some higher order app... twittervision is a good example. As long as twittervision takes the vertical slices as fixed (i.e. they can't easily influence product direction) this is good, and I agree, you can scale much more easily.
However in larger organizations it's usually these high order products that generate the revenue, and you get the situation flipped. If twittervision was able to heavily influence the direction of Google maps, Twitter, OpenID, etc. you'd end up with the same mess you usually do in fortune 500 dev shops. Features demanded from the verticals before they're ready, resulting in high-stress and unpredictable development timelines.
albert
· 1 year ago
Good point -- discipline is required to prevent the internal services from being overloaded with features.
mccv
· 1 year ago
I threw up a quick counterpoint on the enterprise challenges here http://continuations.wenger.us/post/33100418... it's been pretty well received internally. Thanks for the provocation of thought Albert!
albert
· 1 year ago
Did you mean to include a different link? Would love to read the counterpoint.
Enjoyed reading your post. I believe that the issues you raise are among the reasons why large enterprises are being challenged successfully by startups in so many areas.
bfeld
· 1 year ago
Keep going Albert. What are the other vertical slices that you like?
albert
· 1 year ago
I think there are very few generic ones. Etsy would have a pretty different set of internal services from say Wesabe or Twitter. One good way to identify internal services is to think about the overall API for the site or service and look for logical groups. A candidate that might apply to a lot of sites is a 'social graph' service.
Daniel Feygin
· 1 year ago
SOA is typically thought to be for the big boys, but you make an excellent point about service orientation for startups. I tend to think of this as component-based vs. monolithic architectures, rather than vertical vs. horizontal. A service then is just a type of component accessible via network.
alive88
· 1 year ago
...and I read SOA as opportunity through specialization for smaller companies. You can do just a small part of larger processes. and do just that exact function - very specialized - up to your exact capacity.
This also feeds my bias about anti-scale. Why not create a business at limited scale that has the right economics?
andrewbadera
· 1 year ago
You write as though the data access layer and business logic layer and presentation layers just go away -- far from the truth. These concepts are still completely valid in SOA. Your service still encapsulates business logic (in terms of the enterprise) and data access logic, and your clients still encapsulate client-specific business logic, and certainly any and all presentation logic. SOA doesn't remove the necessity nor value of these things -- it simply shifts responsibility, and takes the notion of encapsulation to a whole new level. It's OO growing into distributed computing.
All that said, I love SOA -- I work on an enterprise remoting architecture and numerous web services -- but Service Oriented Architectures still have their own overhead. Who documents the API? Who maintains the service directory? Who communicates between all these disparate teams? Who acts as arbitrator, deciding which team or which service or business user is authoritative? Who decides what represents truly new functionality vs. enhancements to an existing service? Who manages the client side of things, to ensure compliancy with a changing API? With headless clients and processes, you don't always have instant "hey their release broke my implementation" feedback like you do with Twitter or flickr.
SOA is no magic wand. Like all else in code, it has its place, it has pros, it has cons. Yes, it is extremely scale-friendly in terms of delivering capacity, but the more you scale, the more communications and documentation and authority issues become apparent.
Finally, a note on startups and cost of entry: it's essentially nil. You can't possibly make a case for tabula rasa SOA being any more expensive than starting from a clean slate with traditional OO. And, the earlier you start, the more power and flexibility SOA will offer you, and sooner.
albert
· 1 year ago
You are right. I did not mean to imply that these layers disappear, simply that they are encapsulated. They beauty is that the implementation of layers might differ across various internal services, using the technology that's most appropriate for the particular service.
Aneel
· 1 year ago
The problem is no different for many large, established enterprises. Having never figured out how to scale, they developed as fiefdoms of incompatible services, layers, processes, etc. The solution, I think, is the same--a services-api. Let them do wtf they want to internally as long as they conform to the apis. Then you can have a common language for interconnecting bits of your disintegrated enterprise in novel/innovative ways.
All Comments
As important as scaling, are methods for auditing the feed from the api as it flows and joins with other feeds.
A comment on AVC mentions the Financial industry figured this out long ago. We agree - but their budget is in the 100's of millions - each year :)
Twitter API Fail ~ Location
Foremost, our point of view comes from twitter stating they are becoming a “global communication utility”.
... more on our tumblelog, or on twitter dev blog, or via shortened url :) http://tweetip.us/lkrha
by not drafting an advanced specification, twitter failed to insure the integrity of the data. Mobile app developers just started pumping in whatever format/values they desired. This data "breaks" our app, if we are filtering values based on historical analysis as is the case of First Responder / Earthquake / Tornado / Emergency Services or for that matter Market Movement / Breaking News / any realtime anything where we feel Location is integral to our analysis.
There's more on this convo at the twitter google dev blog.
twitter just broke API 4 tons of apps by introducing rate limits 2 unauth calls - way 2 kill app ecology —via @emp
However in larger organizations it's usually these high order products that generate the revenue, and you get the situation flipped. If twittervision was able to heavily influence the direction of Google maps, Twitter, OpenID, etc. you'd end up with the same mess you usually do in fortune 500 dev shops. Features demanded from the verticals before they're ready, resulting in high-stress and unpredictable development timelines.
This also feeds my bias about anti-scale. Why not create a business at limited scale that has the right economics?
All that said, I love SOA -- I work on an enterprise remoting architecture and numerous web services -- but Service Oriented Architectures still have their own overhead. Who documents the API? Who maintains the service directory? Who communicates between all these disparate teams? Who acts as arbitrator, deciding which team or which service or business user is authoritative? Who decides what represents truly new functionality vs. enhancements to an existing service? Who manages the client side of things, to ensure compliancy with a changing API? With headless clients and processes, you don't always have instant "hey their release broke my implementation" feedback like you do with Twitter or flickr.
SOA is no magic wand. Like all else in code, it has its place, it has pros, it has cons. Yes, it is extremely scale-friendly in terms of delivering capacity, but the more you scale, the more communications and documentation and authority issues become apparent.
Finally, a note on startups and cost of entry: it's essentially nil. You can't possibly make a case for tabula rasa SOA being any more expensive than starting from a clean slate with traditional OO. And, the earlier you start, the more power and flexibility SOA will offer you, and sooner.
This seems very close to Aspect Oriented Programming:
http://en.wikipedia.org/wiki/Aspect_oriented