Continuations - Latest Comments in System and Organizational Scaling - Continuations

Re: System and Organizational Scaling - Continuations

albert — Thu, 17 Jul 2008 17:32:07 -0000

Ouch - that's not good that they broke stuff but I understand why they would want rate limits.

Re: System and Organizational Scaling - Continuations

tweetip — Thu, 17 Jul 2008 17:09:08 -0000

Respectively, we submit the beginnings of the firestorm we've politely tried to make you aware of:

twitter just broke API 4 tons of apps by introducing rate limits 2 unauth calls - way 2 kill app ecology —via @emp

Re: System and Organizational Scaling - Continuations

albert — Sat, 12 Jul 2008 22:12:08 -0000

Thanks for clarifying -- I understand this better now.

Re: System and Organizational Scaling - Continuations

tweetip — Sat, 12 Jul 2008 22:08:49 -0000

from a mission critical developer/end user point of view, and twitter being a ---> worldwide comm utility <---

by not drafting an advanced specification, twitter failed to insure the integrity of the data. Mobile app developers just started pumping in whatever format/values they desired. This data "breaks" our app, if we are filtering values based on historical analysis as is the case of First Responder / Earthquake / Tornado / Emergency Services or for that matter Market Movement / Breaking News / any realtime anything where we feel Location is integral to our analysis.

There's more on this convo at the twitter google dev blog.

Re: System and Organizational Scaling - Continuations

albert — Sat, 12 Jul 2008 21:36:04 -0000

I read the post on your tumblelog, but am not sure I understand what the failure in the Twitter API is.

Re: System and Organizational Scaling - Continuations

tweetip — Sat, 12 Jul 2008 16:18:39 -0000

Hi Albert. Most often, our point of view takes awhile to manifest:

Twitter API Fail ~ Location

Foremost, our point of view comes from twitter stating they are  becoming a “global communication utility”.

... more on our tumblelog, or on twitter dev blog, or via shortened url :) http://tweetip.us/lkrha

Re: System and Organizational Scaling - Continuations

alive88 — Tue, 06 May 2008 02:39:10 -0000

...and I read SOA as opportunity through specialization for smaller companies. You can do just a small part of larger processes. and do just that exact function - very specialized - up to your exact capacity.

This also feeds my bias about anti-scale. Why not create a business at limited scale that has the right economics?

Re: System and Organizational Scaling - Continuations

albert — Sat, 03 May 2008 17:32:07 -0000

Enjoyed reading your post. I believe that the issues you raise are among the reasons why large enterprises are being challenged successfully by startups in so many areas.

Re: System and Organizational Scaling - Continuations

Aneel — Fri, 02 May 2008 20:27:17 -0000

The problem is no different for many large, established enterprises. Having never figured out how to scale, they developed as fiefdoms of incompatible services, layers, processes, etc. The solution, I think, is the same--a services-api. Let them do wtf they want to internally as long as they conform to the apis. Then you can have a common language for interconnecting bits of your disintegrated enterprise in novel/innovative ways.

Re: System and Organizational Scaling - Continuations

mccv — Fri, 02 May 2008 16:21:40 -0000

Ergh, of course http://www.themcwongs.com/m...

Re: System and Organizational Scaling - Continuations

albert — Fri, 02 May 2008 16:16:03 -0000

Did you mean to include a different link? Would love to read the counterpoint.

Re: System and Organizational Scaling - Continuations

mccv — Fri, 02 May 2008 15:37:35 -0000

I threw up a quick counterpoint on the enterprise challenges here http://continuations.wenger...... it's been pretty well received internally. Thanks for the provocation of thought Albert!

Re: System and Organizational Scaling - Continuations

albert — Mon, 28 Apr 2008 20:29:15 -0000

You are right. I did not mean to imply that these layers disappear, simply that they are encapsulated. They beauty is that the implementation of layers might differ across various internal services, using the technology that's most appropriate for the particular service.

Re: System and Organizational Scaling - Continuations

albert — Mon, 28 Apr 2008 20:24:51 -0000

Good point -- discipline is required to prevent the internal services from being overloaded with features.

Re: System and Organizational Scaling - Continuations

albert — Mon, 28 Apr 2008 20:22:48 -0000

Agree that APIs need a good approach to versioning for changes that are not backward compatible. Ideally though, most changes to an API make it more powerful without breaking existing apps.

Re: System and Organizational Scaling - Continuations

andrewbadera — Mon, 28 Apr 2008 12:49:40 -0000

You write as though the data access layer and business logic layer and presentation layers just go away -- far from the truth. These concepts are still completely valid in SOA. Your service still encapsulates business logic (in terms of the enterprise) and data access logic, and your clients still encapsulate client-specific business logic, and certainly any and all presentation logic. SOA doesn't remove the necessity nor value of these things -- it simply shifts responsibility, and takes the notion of encapsulation to a whole new level. It's OO growing into distributed computing.

All that said, I love SOA -- I work on an enterprise remoting architecture and numerous web services -- but Service Oriented Architectures still have their own overhead. Who documents the API? Who maintains the service directory? Who communicates between all these disparate teams? Who acts as arbitrator, deciding which team or which service or business user is authoritative? Who decides what represents truly new functionality vs. enhancements to an existing service? Who manages the client side of things, to ensure compliancy with a changing API? With headless clients and processes, you don't always have instant "hey their release broke my implementation" feedback like you do with Twitter or flickr.

SOA is no magic wand. Like all else in code, it has its place, it has pros, it has cons. Yes, it is extremely scale-friendly in terms of delivering capacity, but the more you scale, the more communications and documentation and authority issues become apparent.

Finally, a note on startups and cost of entry: it's essentially nil. You can't possibly make a case for tabula rasa SOA being any more expensive than starting from a clean slate with traditional OO. And, the earlier you start, the more power and flexibility SOA will offer you, and sooner.

Re: System and Organizational Scaling - Continuations

Daniel Feygin — Mon, 28 Apr 2008 12:32:43 -0000

SOA is typically thought to be for the big boys, but you make an excellent point about service orientation for startups. I tend to think of this as component-based vs. monolithic architectures, rather than vertical vs. horizontal. A service then is just a type of component accessible via network.

Re: System and Organizational Scaling - Continuations

IanWilson — Mon, 28 Apr 2008 11:51:24 -0000

Beautiful and succinct, I think this will put a name to many companies pain.
This seems very close to Aspect Oriented Programming:
http://en.wikipedia.org/wik...

Re: System and Organizational Scaling - Continuations

mccv — Mon, 28 Apr 2008 11:46:17 -0000

I think the challenge here is that eventually somebody will want to add a horizontal layer that encompasses multiple vertical slices. Usually this is some higher order app... twittervision is a good example. As long as twittervision takes the vertical slices as fixed (i.e. they can't easily influence product direction) this is good, and I agree, you can scale much more easily.

However in larger organizations it's usually these high order products that generate the revenue, and you get the situation flipped. If twittervision was able to heavily influence the direction of Google maps, Twitter, OpenID, etc. you'd end up with the same mess you usually do in fortune 500 dev shops. Features demanded from the verticals before they're ready, resulting in high-stress and unpredictable development timelines.

Re: System and Organizational Scaling - Continuations

tweetip — Mon, 28 Apr 2008 11:38:03 -0000

We're big fans of API's - unfortunately, they do break. Each rev of twitter has had an api glitch - often an issue not easily detected.

As important as scaling, are methods for auditing the feed from the api as it flows and joins with other feeds.

A comment on AVC mentions the Financial industry figured this out long ago. We agree - but their budget is in the 100's of millions - each year :)

Re: System and Organizational Scaling - Continuations

albert — Mon, 28 Apr 2008 09:35:02 -0000

I think there are very few generic ones. Etsy would have a pretty different set of internal services from say Wesabe or Twitter. One good way to identify internal services is to think about the overall API for the site or service and look for logical groups. A candidate that might apply to a lot of sites is a 'social graph' service.

Re: System and Organizational Scaling - Continuations

bfeld — Mon, 28 Apr 2008 09:17:19 -0000

Keep going Albert. What are the other vertical slices that you like?