How far can you go with GET, PUT, POST, DELETE? Much of the programming world asks the question. Here in one place are the complete collected episodes from Ted Neward’s and Adrian Trenaman’s recent conversation on the matter of SOAP and POX.
The interpid Neward suggests objects and XML have just as deep an impedance mismatch as objects and relational data do. REST is a good architectural style, admits Adrian, but he notes that SOAP, since it’s transport-agnostic, would work even if you wanted to send the message over other transports beyond HTTP, like JMS messages. Check it out!
This time in San Francisco for the JavaOne 2007 conference, Adrian Trenaman deliberately tracked Ted Neward down at one of the innumerable “after-parties”–this one hosted by some company whose name sounded something like “oogle”–and started up another conversation, this time on the growing debate between “SOAP” and “Plain Old XML”, or “POX”. Once again, TheServerSide was able to capture the discussion, despite the best efforts of several large men in black suits and sunglasses who kept interrupting the early parts of the conversation with brusque challenges of “Continue or Cancel?”. Rumor has it these “G-Men” are part of a new multinational intelligence organization, but that’s another story for another day.
(Editor’s note: Again, as with the prior conversation on contract-first and code-first design, both men had consumed many of the available beverages of questionably moral character, and were still a touch irritated at the “G-Men” trying to display advertising in the background of their conversation. Readers unaccustomed to raucous debate are, again, forewarned.)
Ted: (in his best “Rocky Balboa” voice) Yo, Adrian!
Adrian: Har, har.
Ted: Sorry, man, couldn’t resist.
Adrian: Hey, did you see that our conversation in London was recorded somehow?
Ted: Yeah, I saw that–did you do that?
Adrian: No, I thought you did. Weird. Oh, here’s another waiter with wine. You want one? (Mumbled deep voices interrupt the tape here, sounding vaguely like “Please talk into the lapel” and “Continue or Cancel?”)
Adrian: So I wanted to ask you about this SOAP vs. REST thing that’s going around.
Ted: Oh, boy, here we go again.
Adrian: (Laughs) What, you didn’t have fun last time?
Ted: (Laughs) OK, SOAP vs REST, though I think we’re better off characterizing it as a “SOAP vs POX” debate.
Adrian: Ah yes - Plain Old XML. Funny that. You’d have thought that soap would be just the cure for the pox.
Ted: Yup, strange but true: there is actually a whole lot of Plain Old XML out there -in other words, just ignore the SOAP part of things and just send XML back and forth with nothing surrounding it.
Adrian: See, that’s just it. I’ve been musing about this for the last few days. I remember the pain in the early days of SOAP, when the spec included it’s own “SOAP-encoding”…
Ted: Agh! Don’t ever bring that name up to me again!
Ted: SOAP-Encoding was such a POS, that…
Adrian: POS? Plain Old String?
Ted: No, Piece of Sh*t.
Adrian: (Laughs) Sorry, thought we’d coined a new acronym there.
Ted: Anyway, as near as I can tell, SOAP-encoding was an almost-afterthought part of the spec, designed to be a quick-and-dirty way of transcribing objects into XML and back again, and had holes large enough to drive a Mack truck through. When I’d heard the WS-I guys deprecated SOAP-encoding entirely, I danced the HappyHappyJoyJoy dance for hours.
Adrian: I hear ya. I was still young then. I had more hair. Hmmm. Hair loss due to Interop problems. Can I claim that on health insurance? (Ted sighs and does his impression of a drum role at a comedy gig).
Adrian: I remember pain around bloated payload, and lots of interop issues around arrays. You’d think they could have figured it out…
Ted: Yeah - but there’s some even deeper issues here, one being that objects and XML have just as deep an impedance mismatch as objects and relational data do.
Adrian: You’ve mentioned that before Ted - sure there’s a mismatch. But hey, let’s not throw the baby out with the bath water!
Ted: On the contrary, lets! Look - consider this simple object model, represented by this class
Adrian: Is that Java or C#?
Ted: There’s a difference?
Adrian:Point taken. OK, you have this class, now what?
Ted: Well, OK, assume I’ve created a couple of objects, like so:
Adrian: Who’s Jay and Libby?
Ted: They run the No Fluff Just Stuff shows. Nice folks.
Ted: Anyway, so now I want to pass Jay as a parameter in an XML service call. What should the corresponding XML look like? Before you answer, let me run you through the usual suggestions. The first says that fields just map to child elements, and you follow the references accordingly.
Adrian: OK, sounds good, but…
Ted: I know, but hang on. If I do that, we run into a problem pretty quickly, right?
Adrian: Ouch. I hear ya. No way to encode references.
Ted: Yep. Because XML isn’t designed to be recursive in nature–it’s just a tree-based data structure–it has a really hard time dealing with these kinds of recursive object models, which happen all the time in object designs.
Adrian: So what’s this got to do with SOAP-encoding?
Ted: Well, the SOAP guys initially tried to solve this problem, by creating an encoding mechanism on top of the XML structure–
Adrian: The Infoset.
Ted: –right, on top of the Infoset, so it would look something like this:
which, of course, isn’t valid XML, but to them, that was OK, because they’d surround this with an outer element anyway.
Adrian: So what was wrong with it?
Ted: Well, for starters, how do you write an XPath query to find the spouse of the spouse of the person named Jay?
Adrian: Um, I can’t think of one.
Ted: Right–that’s because this is encoded on top of the Infoset, and all the various X-languages and tools are designed to work with the Infoset, not on top of it. Second, imagine what happens if we have a graph of over a hundred objects referenced from Jay? What happens when he gets passed in?
Adrian: Oh, yuck–they’d all get serialized in, don’t they?
Ted: Yep, and you end up passing a lot more data across the wire than you’d originally intended or wanted.
Adrian: So was that it?
Ted: Oh, no, there was more–what happens if you get an error in transmission, such as something like this?
Adrian: Well, I’d imagine you’d find this out when you try to recreate the object graph, right? You can’t create a Pet object and store it into a Person reference, I would think.
Ted: Sure, but (a), you’re only going to get this error after you’ve parsed the entire XML graph and instantiated a bunch of those objects, and (b) what about those languages that aren’t object-oriented?
Adrian: You mean there are languages that aren’t object-oriented? Seriously?
(Both men laugh)
(Editor’s note: At this point, one of the G-men stepped in, mumbled “Continue or Cancel?”, and while Ted and Adrian discussed programming languages for a while, our agent was forced to disguise himself as a plastic plant.)
Adrian: So we’re happily agreed! The SOAP-encoding is dead, may it rest in relative discomfort.
Ted: Yup! It was by far and away the biggest reason for SOAP incompatibility in the early days. If I remember the stats right, the SOAPBuilders website used to track compatibility of SOAP toolkits, and if you picked two SOAP toolkits completely at random, you had about a 50/50 chance of them working together.
Adrian: Right, but that’s my point: in the early days, SOAP was responsible for a lot of the incompatibility mismatches, but we’re past that now, right?
Ted: Yep. Cheers to that.
(Sounds of glasses clinking.)
Adrian: So that’s all in the past: SOAP-encoding has long since bitten the dust, we’ve all moved to using literal XML data, defined by schema. What that means, then, is that after removing the SOAP encoding stuff, SOAP really boils down to just two things: one, a wrapper around your XML payload, called Body, and optional headers to the message, to carry out-of-band information, like security credentials, transaction IDs, session IDs, or whatever.
Ted: Yep. A buddy of mine once remarked that the SOAP specification is the specification that doesn’t really specify anything.
Adrian: (Laughs) So, if I can borrow your cocktail napkin for a bit–
(Editor’s note: Again, our agent, cleverly disguised as a cocktail napkin, was able to capture the text of the example.)
Adrian: –it looks something like this:
So, let’s say I want to send some XML from me to you. I can just drop the XML into the Body element, and that’s it. If I want, I can put fancy headers in the Header element with security, reliability or transaction information, as per the multitude of WS-Everything specs. Alternatively, I can just put my own custom headers in there: maybe I’ve got my own, in-house military strength encryption headers, or my own session management scenario.
Ted: Though I’d strongly suggest you not try to create your own encryption. Bad juju there, baby.
(Sounds of glasses clinking again.)
Adrian: So, here’s my question to you, why do people oppose SOAP so much?
Ted: Well…. That’s actually kinda hard for me to argue effectively, because I tend to side with you on this one: I like SOAP, simply because at the end of the day, it’s just a messaging format designed for framing and extensibility. Once we pull the SOAP-encoding bits out of the specification, there’s really nothing all that complex about it anymore.
Adrian: Aha! We agree!
Ted: Shush! Don’t let anybody hear you say that! You’ll lose respect!
Ted: Seriously, though, I’ve heard a couple of objections, which I’m happy to argue with you about, if you like.
Adrian: Fire away.
Ted: Well, the first one that’s commonly tossed off is that SOAP is too complex, and REST, or perhaps we’re better off calling it POX-over-HTTP, is a much simpler way to go and therefore much more approachable.
Adrian: Hmm. I’d have to argue that this is a pretty weak objection, given the example I just wrote a few minutes ago. Assuming no headers, you’re talking two extra XML elements around your XML payload data, and that’s hardly an unmanageable increase in complexity, if you ask me.
Ted: I’d have to agree, particularly if you use XPath to extract the data from a parsed Infoset structure or model it using JAXB or XMLBeans or one of the other toolkits.
Adrian: I’ve heard people accuse SOAP of being slow - but that’s hardly a credible argument if you’ve bought into XML in the first place!
Ted: Yeah - honestly I’ve never benchmarked the XML parser against a binary parser, and in general I don’t care much, because we’re generally talking about out-of-process communication anyway, and the time spent marshaling and unmarshaling the data to and from XML is going to get lost in the noise of sending the data across the wire.
Adrian: So what’s the principal argument, then?
Adrian: Bless you.
Ted: (Laughs) No, thanks anyway. YAGNI is the acronym agile folks love to toss at things they think are too complex or overengineered: You Ain’t Gonna Need It. It’s a term that basically suggests that if it’s not something we need right away, then don’t bother building it in, because You Ain’t Gonna Need It.
Adrian: They don’t believe in evolving systems?
Ted: No, they believe that engineers have this tendency to gold-plate stuff on the murky grounds of “I know that we’ll need this someday”. Then, about half the time, customer needs change, or projected phases to the application don’t happen, and they never end up using that stuff they built in–
Adrian: –which makes all that work they did completely meaningless.
Ted: Right, because, as it turned out, they didn’t need it in the end.
Adrian: Hmmm - yeah. Still feels week though - I really can’t accept that using a SOAP envelope is dragging us into The Marsh of Unwarranted Flexibility.
Ted: Speaking of which, let’s not get too bogged down ourselves. I believe I spot the open bar in the corner, and the other phrase that comes to mind is IAGNAGOW.
Ted: I am gonna need another glass of wine. You want one?
(Editor’s note: At this point, our agent, cleverly disguised in no disguise at all but looking like an attendee at the party, was ejected from the party for being too quiet and not drinking enough.)
(Editor’s note: This is the third part of a continued series of transcriptions from a conversation secretly recorded at an “after” party in San Francisco thrown by a company who shall remain nameless but rhymes with “oogle” during the JavaOne 2007 conference between Ted Neward and Adrian Trenaman, who were discussing the SOAP-vs-REST debate.)
Ted: Here you go.
Adrian: Thanks. Cheers. (Sounds of clinking glasses.) Not bad. What’d the bartender say this was, again?
Ted: I think they’re calling it G-Wine. It’s in beta. See the ad on the glass?
Adrian: Neat. So you were saying something about YAGNI?
Ted: Oh, right. The REST/POX folks often argue that SOAP’s headers are an unnecessary complication, since HTTP already provides facilities for communicating data out-of-band like SOAP does, so therefore if I never needed to send some of those custom headers, HTTP already provides me what I need.
Adrian: True… though I admit I’m surprised that everybody assumes that we always want to send the data over HTTP.
Ted: That one catches me off-guard, too. Personally, I don’t like to run an HTTP server on machines that aren’t facing the outside world, particularly when there are faster protocols for communicating data within a subnet, or when I want semantics that HTTP can’t provide, like offline capabilities or recovery.
Adrian: You sound like you’re a big JMS fan!
Ted: Of any messaging system, in fact. JMS is just one of them. Funny thing is, when you look at SOAP, HTTP, JMS, and other messaging engines, you see a really close parallel in all of them: payload and headers.
Adrian: All the world’s communication is just messages, back and forth. We are but packets in the Internet of life.
Ted: (Laughs) Seriously, though, every RPC system I’ve ever seen is basically nothing more than a request/response message pair, with the client blocking on the response after sending the request. That’s part of the reason I don’t like WSDL that much, to be honest–it builds an unattractive abstraction on top of the messages I want to get hold of.
Adrian: Well, I dunno - as you know I’m well down with funky WSDL stuff. But the debate of either WSDL+SOAP vs XSD+XML is almost religious at this point - I have a feeling that people choose a path and then just stick to it. I’ve seen a lot of folks recently reverting to using XML, particularly in a RESTful way.
Ted: Exactly - but then that’s the interesting thing. If you read the original Fielding thesis, or some of the later summarizations by Fielding, you realize that what he’s really arguing for is a communication architecture for a distributed hypermedia system that can scale anarchically.
Adrian: Ah yes - Anarchy. No central point of control: God Bless the Internet.
Ted: In a nutshell. Problem is, REST assumes that you want to carry all of the state in the payload itself, and for a modern enterprise system, or, hell, even for a game, that’s not always a safe assumption. REST has no sense of “session” because of that same reason: all you need to know about what happens next is contained in the document that’s sent back to you. Hence the reason cookies were never a part of the HTTP specification and why Fielding hates them so badly–they cramp the scalability of the Web server.
Adrian: So all those people arguing REST for enterprise systems….
Ted: … are basically arguing something they don’t understand, yep. In fact, if you read the dissertation, he’s arguing for a small vocabulary of verbs–
Adrian: –you mean like GET, PUT, POST, DELETE–
Ted: Yep, and that the data is accompanied by metadata describing the data and providing opportunities for extension–
Adrian: –such as the headers that accompany HTTP documents–
Ted: Yep, which sounds an awful lot like–
Adrian: SOAP! Ta da!
Ted: No kidding!
Adrian: Wow. That calls for another drink or four. Now. Lets prove that black == white…
(Editor’s note: The conversation paused while Adrian moved off to the bar to get a bottle of wine–a 2007 House Red–with which to continue the conversation. Our agent, stunned at the implications, immediately moved to secure the remainder of the discussion as they moved out into the hall outside.)
Adrian: That’s pretty heavy stuff.
Ted: REST is a good architectural style. We just need to recognize that we’re not always exchanging hypermedia documents.
Adrian: But the nice thing with SOAP is, since it’s transport-agnostic, it would work even if you wanted to send the message over other transports beyond HTTP, like JMS messages.
Ted: Or over God’s original messaging system, SMTP/POP3.
Adrian: Or ICMP, or Jabber, or… the possibilities are endless!
Ted: Yep. And the beautiful thing is, all those WS-* specs operate on the message, not the underlying transport, which is important for things like security. The encryption is in the message itself, so even if the message is resting someplace on a server waiting to be sent on to another server, the data’s not floating around in an unencrypted format waiting to be sent onwards. Combine that with the fact that HTTPS only works point-to-point, not across intermediary machines, and I think you come to a point where you realize that the overhead that SOAP adds to the XML you want to send back and forth is pretty miniscule compared to the future-proofing you buy with it, particularly given that vendors are slowly coming out with pre-canned WS-* stacks that I don’t have to build myself.
Adrian: So you think everybody should use SOAP?
Adrian: (Laughs) Not ready to go that far? Neither am I. I’ve run across applications where, quite frankly, SOAP just isn’t required. For example, if you’re a pure JMS house, then you can make use of JMS transport attributes to cover security, transactionality and addressing. Or, if you’re sending XML over HTTP and need it to be secure, then you could just secure the transport using HTTPS and then send any additional credentials using HTTP Basic Authentication.
Ted: But that only works if you’re going point-to-point, remember.
Adrian: Oh, sure. But in general, I’m a big fan of parsimony: if there genuinely is no use for SOAP in an application, then we shouldn’t force it in. I’m all in favour of conscientious objectors. I think it’s important though every now and again to revisit design assumptions. SOAP can still be hugely helpful in existing POX applications: for a start, you’ve got the concept of a SOAP fault: very handy for dealing with bad payload in a polite way.
Ted: And an extensible way. Rather than having to rely on a finite set of error codes defined by the HTTP protocol, which don’t always map over to your application’s needs, you get to extend and describe the errors in an open–but arguably more complicated–way. Plus, your multi-transport discussion comes into play here, too. If you’re sending messages in multiple hops across different transports (HTTP, JMS, AMQP, and so on) then it’s good to put the headers in the message payload, rather than depend on features that are present in one transport and not in the next.
Adrian: You realize, of course, that if this conversation ever got out of this room, you’d probably be lynched at the next conference you go to?
Ted: That’s why I’m making very sure that we’re not standing next to any waiters, floral arrangements, CIA agents, ceiling fans, or attendees as we do this.
(Editor’s note: At this point, our agent, cleverly disguised as a company-branded blinking-light pin resting on Ted’s T-shirt, Gmailed the conversation back to his handlers at the secret headquarters of TheServerSide, completing yet another dangerous mission.)