5 ms·

I'm being asked to create webservices so other developers don't do direct database access and their stuff wont break when things change. How do you manage that?

didip 16y ago
If you use MySQL, slap this bad-boy(http://code.nytimes.com/projects/dbslayer http://code.nytimes.com/projects/dbslayer) between the app and the database. The payload is JSON over HTTP.
dansingerman 16y ago
Designing an API is a skill in itself, but I think rule 1 is "Don't use SOAP" JSON over REST seems to be flavour of the month FWIW.
- roc 16y ago
  "Flavour of the month" might be a bit harsh. It certainly is popular these days, so it's not wholly inappropriate. But it's worth noting JSON/REST is popular with, and largely because of, people who've learned the painful lessons of earlier messes like SOAP. It's not (just) a hot new thing used by new faces who've never really known any other way.
  - dansingerman 16y ago
    I actually didn't mean to disparage JSON over REST in any way. In fact I really like it. I only qualified the statement as technologies change quickly (only 12 months age we would probably have been talking about XML over REST) and things will no doubt change again.
- mainguy 16y ago
  JSON or XML over REST is certainly simpler and gaining popularity. The reason it isn't being broadcast on EWeek/etc is IBM, Microsoft and the rest of the tool players haven't figured out how to sell you a tool that locks you into their proprietary stack. A key point the writer alludes to (and an IMPORTANT one) is that SOAP was hyped by tool vendors because it game them something to SELL you. There's really nothing to SELL with RESTful stuff, it just works and is inherent to the internet, therefore tool vendors aren't exactly pumped up about selling you anything.
cturner 16y ago
One of several great motivations for that approach is that you want the power to refactor the database later without being limited by the fact that you have seven applications from seven different divisions of the company all hooked up to a single schema design. You want to be able to moderate through a business logic layer. Many businesses wind up in this situation by chance, and then have a very painful journey out of the casual schema they created when they were a small business into something which matches their larger business. Unfortunately, IMO, supplying a good answer to that question is the 'northwest passage' of software development. I've spent years at it on and off. I can create such a layer, but the solutions I've come up with are unwieldy from the perspective of the business layer programmer, which is an inadequate solution. -- If I have another attempt, I'll try to create something which would look to the observer quite a lot like visual basic, but where instead of constructing a Windows forms, you're constructing a state, which consists of a set of forms and a variable store. The datatypes for the fields would be something like this: Readonly Boolean Text (untyped) SingleSelect (conceptually like html dropdown) MultiSelect (like checkbox) ExposeObject (the object is tagged, client can bookmark these these objects in its session) ReferenceSocket (the socket indicates the tags it accepts, exposes compatible tags, connection point for bookmarked objects with compatible tags) Link (like html link) Action (like html submit button) A client API then migrates this with a query language that navigates the states. An example of a dumb session that knows nothing of the app to which it's connecting, and which is rendering a UI on the fly: # Remote application knows nothing list all # It gets back a description of the current state, 'login' # Remote application logs in to session assert form is login set username wizard_2 set password whatever action login list all # Remote application has control again # Remote application wants to know what operations are available assert form is portal list actions # Remote application decides it wants to go to person assert form is portal link "manage person" list actions # Remote app decides to add a person assert form is "manage person" link new list all # Shows fields and actions assert mode is new set firstname Wiz set lastname Ard set age 40 select male from singleselect action save # Application throws an exception with details # Client app assert mode is new set data_of_birth 08081971 action save list form Just as SQL is a flexible language for a relational model, this language is a flexible language for an interactive state graph. A client progrmamer who know the state mapping of the app could write tighter instructions than this. That client app can be anything: desktop app, unit test framework, web app, direct rpc over http. Conceptually the situation of states, objects and actions is very similar to a text adventure, but I'd been working on it for years before I realised this. Every time the state progresses from one to the next, it keeps a copy of the previous state in memory. This logic layer has the database underneath it. The DB is now abstracted away from the user-facing application. Major benefits of this approach: - You can design the application with almost no concern to user interface. Then you can create a streamlined user interface later that leverages the correct business logic. You can separate the costing for both projects. That's how I came to the problem. It would revolutionise software consulting. - You can wrap a dumb client interface around the engine. Want your entire business logic exposed to the web, securely? Or RPC? Or mobile? Or for the blind? Done, with literally zero extra effort. - If we had a relational database that had version numbers for its internal state (think git model), you could write applications that could unwind and rewind. Takes a lot of memory, but you could store the states in a nosql database. -- OK. I'll try to answer your question now with the best I know of current techniques. Things using current technology that somewhat address the question you asked: - Filter all chat with the database through stored procedures. Problems with this: (1) non-stateful, you don't have the user's application session; (2) stored procedure languages are a pain to work with. This is the sort of solution you'll find most often in large organisations because you can hire a single known skillset (DBA) who knows both stored procedures and schemas. It's awful but it works. - Put a HTTP layer between the database and the application. Do all interaction with the database through this. In the payload that users send in they could (1) name the action they want to do such as create person and (2) supply the arguments they should send in for that business case. Basically, here you're doing stored procedures but in language of your choice. Latency is higher than with real stored procedures. - Have a rule that all application programmers integrate with the application using Apache Cayenne or Hibernate or something similar as an object-relational modeler, in a strongly-typed language. Insist on use of Data Access Object pattern (see Fowler) for any original entrypoint to the object graph. Now when you refactor your database, regenerate your schema, and use the static typing checking in Java to rapidly adjust your application to the changed schema. You'd need to get all client apps to do this, but it's a lot more rapid than if you've got application users using SQL.
Rabidgremlin 16y ago
mmmm, I'd be very wary of doing this. This kinda of "data" or "infrastructure" level service often devolves into a service that can accept SQL or lands up exposing the oddities of the DB anyway. Because they are low-level they also tend to be very chatty and performance goes right out the window. They probably should be an anti-pattern. If you image a pyramid of service types: Business Process Business Operation Component Infrastructure/Data Then you want to shoot for building "Business Operation" type services. These give you the maximum reuse in an enterprise as they encapsulate a useful atom of business functionality. They hide the complexity of an enterprise's IT systems from consumers and typically use semantics that are business not IT system specific. For example CreateOrder, CreateCustomer etc... Component services are the other useful one. Typically used to wrap a system with web services. This eases integration and allows for aggregation of these services into Business Operation services. For instance the CreateCustomer service above might call CreateCustomer in System A, RegisterMember in System BB and write a record into a the Users table of System C (sometimes its impossible/cheaper not to use web services). As for as the consumer (the helpdesk app, the self service portal etc.) is concerned they just created a Customer within the business and they don't need to know anything about Systems A, B and C.
mattmanser 16y ago
You can't, you're being asked the impossible. Think like this, you need to add the compulsory field of PO Number to your invoice table. It wasn't compulsory before. How do you not break everything? It's impossible. Or, even worse; Projects can now have multiple categories. You change your code so all old requests still work, but suddenly half your systems give users the flexibility to add multiple categories and half don't. Why's that worse? Because you suddenly have people creating projects in one system that only allows one category and then logging into another to add the extra categories because IT doesn't actually have to fix it. Total user nightmare. And to top it all off all, and this is the actual killer, developers are having to use your never quite finished system which throws weird errors that people can't google like 'unable to add project', instead of 'violation of Foreign Key Constraint, FK_Project_User, on table Project' which tells them exactly the error. Use standard DB access because there's a wealth of help out there, your inhouse system can't compare unless you're a big company. Basically you've been asked to fix one of the fundamental problems of why enterprise programming is hard.
- wizard_2 16y ago
  I don't buy this, you have to plan the versions of your "api" if I'm going to change a field into a required field I have to have a plan for clients who don't supply it. You skip all the business and validation logic with direct db access. As for error codes, it doesn't seem non trivial to explain why your validation failed.
vyrotek 16y ago
WCF Data Services - http://msdn.microsoft.com/en-us/data/bb931106 http://msdn.microsoft.com/en-us/data/bb931106
btilly 16y ago
Google's internal solution is to encode data using http://code.google.com/p/protobuf/ http://code.google.com/p/protobuf/ and then to send it between processes.

didip 16y ago

If you use MySQL, slap this bad-boy(http://code.nytimes.com/projects/dbslayer http://code.nytimes.com/projects/dbslayer) between the app and the database. The payload is JSON over HTTP.

dansingerman 16y ago

Designing an API is a skill in itself, but I think rule 1 is "Don't use SOAP" JSON over REST seems to be flavour of the month FWIW.

roc 16y ago
"Flavour of the month" might be a bit harsh. It certainly is popular these days, so it's not wholly inappropriate. But it's worth noting JSON/REST is popular with, and largely because of, people who've learned the painful lessons of earlier messes like SOAP. It's not (just) a hot new thing used by new faces who've never really known any other way.
- dansingerman 16y ago
  I actually didn't mean to disparage JSON over REST in any way. In fact I really like it. I only qualified the statement as technologies change quickly (only 12 months age we would probably have been talking about XML over REST) and things will no doubt change again.
mainguy 16y ago
JSON or XML over REST is certainly simpler and gaining popularity. The reason it isn't being broadcast on EWeek/etc is IBM, Microsoft and the rest of the tool players haven't figured out how to sell you a tool that locks you into their proprietary stack. A key point the writer alludes to (and an IMPORTANT one) is that SOAP was hyped by tool vendors because it game them something to SELL you. There's really nothing to SELL with RESTful stuff, it just works and is inherent to the internet, therefore tool vendors aren't exactly pumped up about selling you anything.

cturner 16y ago

One of several great motivations for that approach is that you want the power to refactor the database later without being limited by the fact that you have seven applications from seven different divisions of the company all hooked up to a single schema design. You want to be able to moderate through a business logic layer. Many businesses wind up in this situation by chance, and then have a very painful journey out of the casual schema they created when they were a small business into something which matches their larger business. Unfortunately, IMO, supplying a good answer to that question is the 'northwest passage' of software development. I've spent years at it on and off. I can create such a layer, but the solutions I've come up with are unwieldy from the perspective of the business layer programmer, which is an inadequate solution. -- If I have another attempt, I'll try to create something which would look to the observer quite a lot like visual basic, but where instead of constructing a Windows forms, you're constructing a state, which consists of a set of forms and a variable store. The datatypes for the fields would be something like this: Readonly Boolean Text (untyped) SingleSelect (conceptually like html dropdown) MultiSelect (like checkbox) ExposeObject (the object is tagged, client can bookmark these these objects in its session) ReferenceSocket (the socket indicates the tags it accepts, exposes compatible tags, connection point for bookmarked objects with compatible tags) Link (like html link) Action (like html submit button) A client API then migrates this with a query language that navigates the states. An example of a dumb session that knows nothing of the app to which it's connecting, and which is rendering a UI on the fly: # Remote application knows nothing list all # It gets back a description of the current state, 'login' # Remote application logs in to session assert form is login set username wizard_2 set password whatever action login list all # Remote application has control again # Remote application wants to know what operations are available assert form is portal list actions # Remote application decides it wants to go to person assert form is portal link "manage person" list actions # Remote app decides to add a person assert form is "manage person" link new list all # Shows fields and actions assert mode is new set firstname Wiz set lastname Ard set age 40 select male from singleselect action save # Application throws an exception with details # Client app assert mode is new set data_of_birth 08081971 action save list form Just as SQL is a flexible language for a relational model, this language is a flexible language for an interactive state graph. A client progrmamer who know the state mapping of the app could write tighter instructions than this. That client app can be anything: desktop app, unit test framework, web app, direct rpc over http. Conceptually the situation of states, objects and actions is very similar to a text adventure, but I'd been working on it for years before I realised this. Every time the state progresses from one to the next, it keeps a copy of the previous state in memory. This logic layer has the database underneath it. The DB is now abstracted away from the user-facing application. Major benefits of this approach: - You can design the application with almost no concern to user interface. Then you can create a streamlined user interface later that leverages the correct business logic. You can separate the costing for both projects. That's how I came to the problem. It would revolutionise software consulting. - You can wrap a dumb client interface around the engine. Want your entire business logic exposed to the web, securely? Or RPC? Or mobile? Or for the blind? Done, with literally zero extra effort. - If we had a relational database that had version numbers for its internal state (think git model), you could write applications that could unwind and rewind. Takes a lot of memory, but you could store the states in a nosql database. -- OK. I'll try to answer your question now with the best I know of current techniques. Things using current technology that somewhat address the question you asked: - Filter all chat with the database through stored procedures. Problems with this: (1) non-stateful, you don't have the user's application session; (2) stored procedure languages are a pain to work with. This is the sort of solution you'll find most often in large organisations because you can hire a single known skillset (DBA) who knows both stored procedures and schemas. It's awful but it works. - Put a HTTP layer between the database and the application. Do all interaction with the database through this. In the payload that users send in they could (1) name the action they want to do such as create person and (2) supply the arguments they should send in for that business case. Basically, here you're doing stored procedures but in language of your choice. Latency is higher than with real stored procedures. - Have a rule that all application programmers integrate with the application using Apache Cayenne or Hibernate or something similar as an object-relational modeler, in a strongly-typed language. Insist on use of Data Access Object pattern (see Fowler) for any original entrypoint to the object graph. Now when you refactor your database, regenerate your schema, and use the static typing checking in Java to rapidly adjust your application to the changed schema. You'd need to get all client apps to do this, but it's a lot more rapid than if you've got application users using SQL.

Rabidgremlin 16y ago

mmmm, I'd be very wary of doing this. This kinda of "data" or "infrastructure" level service often devolves into a service that can accept SQL or lands up exposing the oddities of the DB anyway. Because they are low-level they also tend to be very chatty and performance goes right out the window. They probably should be an anti-pattern. If you image a pyramid of service types: Business Process Business Operation Component Infrastructure/Data Then you want to shoot for building "Business Operation" type services. These give you the maximum reuse in an enterprise as they encapsulate a useful atom of business functionality. They hide the complexity of an enterprise's IT systems from consumers and typically use semantics that are business not IT system specific. For example CreateOrder, CreateCustomer etc... Component services are the other useful one. Typically used to wrap a system with web services. This eases integration and allows for aggregation of these services into Business Operation services. For instance the CreateCustomer service above might call CreateCustomer in System A, RegisterMember in System BB and write a record into a the Users table of System C (sometimes its impossible/cheaper not to use web services). As for as the consumer (the helpdesk app, the self service portal etc.) is concerned they just created a Customer within the business and they don't need to know anything about Systems A, B and C.

mattmanser 16y ago

You can't, you're being asked the impossible. Think like this, you need to add the compulsory field of PO Number to your invoice table. It wasn't compulsory before. How do you not break everything? It's impossible. Or, even worse; Projects can now have multiple categories. You change your code so all old requests still work, but suddenly half your systems give users the flexibility to add multiple categories and half don't. Why's that worse? Because you suddenly have people creating projects in one system that only allows one category and then logging into another to add the extra categories because IT doesn't actually have to fix it. Total user nightmare. And to top it all off all, and this is the actual killer, developers are having to use your never quite finished system which throws weird errors that people can't google like 'unable to add project', instead of 'violation of Foreign Key Constraint, FK_Project_User, on table Project' which tells them exactly the error. Use standard DB access because there's a wealth of help out there, your inhouse system can't compare unless you're a big company. Basically you've been asked to fix one of the fundamental problems of why enterprise programming is hard.

wizard_2 16y ago
I don't buy this, you have to plan the versions of your "api" if I'm going to change a field into a required field I have to have a plan for clients who don't supply it. You skip all the business and validation logic with direct db access. As for error codes, it doesn't seem non trivial to explain why your validation failed.

vyrotek 16y ago

WCF Data Services - http://msdn.microsoft.com/en-us/data/bb931106 http://msdn.microsoft.com/en-us/data/bb931106

btilly 16y ago

Google's internal solution is to encode data using http://code.google.com/p/protobuf/ http://code.google.com/p/protobuf/ and then to send it between processes.