• If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • You already know Dokkio is an AI-powered assistant to organize & manage your digital files & messages. Very soon, Dokkio will support Outlook as well as One Drive. Check it out today!

View
 

DataStore About

Page history last edited by Remy Munch 13 years, 10 months ago
The DataStore framework in SproutCore manages structured data in your applications.  You will usually use it to build some or all of the model layer in your application.
 
In particular, you can use the DataStore framework to automatically:
 
  • Read and write data from/to your server.
  • Manage relationships between objects in your data model
  • Control when changes propagate to the rest of your application
  • Roll back or discard changes if you no longer want them
 
The DataStore also contains built in support for loading data from fixtures which means you can often develop your SproutCore application independently from your server, allowing you to move twice as fast when working in a team.
 

When To Use The DataStore

 
You should use the DataStore framework anytime you need to work with structured content such as users, accounts, contacts, events, transactions, and so on.  Essentially anything you would store in a modern database would be a good fit for the this framework.
 
Although almost every application you write can benefit from using the DataStore, it is not required to build a SproutCore application.  For example, if you are building a simple app such as a login page, you may benefit from leaving this framework out.  
 
Additionally, the DataStore framework is not designed to help you manage document-oriented data such as reports, help files, XML documents, etc.  If your project involves document oriented data, consider using the DataStore to manage high-level records and metadata and then work with the documents on your own.
 

DataStore At a Glance

 
Before we dig into the specifics of how to write applications using the DataStore, it’s useful to get a quick overview of all the pieces you will encounter in this framework and how they fit together.  The section is going to step through all of the different pieces of a typical app using the DataStore.  The diagram below shows you how all of these pieces fit together.
 
 

JSON Data Hashes

 
The first thing you need to know about the DataStore is that it is built around managing data hashes.  A data hash is defined as a set of simple key/value pairs like you would find in a JSON document.  The keys are always strings.  The values are simple data types such as a String, Number or Boolean.  Values may also contain Arrays or other data hashes, though built-in support for these types is less robust so you’ll end up writing more code on your own.
 
A data hash must never contain any data type that cannot be stored directly in a JSON document.  For example, you cannot use an SC.Object, SC.Set, or SC.IndexSet as a data hash value.  The reason is that data hashes are often directly added to a JSON document and then serialized to send it back to the server.  (You can serialize to XML or other formats as well, but these formats have similar restrictions.)  Since JSON only allows certain primitive types in documents, you can only use those same types in data hashes.
 
This is not to say that you cannot use any complex object types in your data model.  In fact you can.  As we’ll see later, the DataStore contains a sophisticated system to let you map data model values back to simple data hashes.  It’s just important to understand from the beginning that fundamentally everything you work with in the DataStore boils down to a simple JSON data hash.
 

The Store

 
Data hashes you load into memory are kept in a single in-memory database called the store.  The store acts as a central coordinator for all of your data.  It keeps track of which data hashes have been modified, deleted, or created and can also perform searches to return data matching particular constraints.  It also talks to your server layer to handle loading data and committing changes back.
 
Usually each application you create will have only one primary store.  In some cases, such as when you have loadable modules with their own independent data sets, they may each create their own store.  Usually however, your application will work best if you have only one store to keep all of your data in it.
 
When you generate a new application, by default, SproutCore creates a new store for you at {AppName}.store.  You can find this line in the core.js file for your app.  You will usually modify this line only when you are ready to switch from using Fixtures to talking to a real server.
 

Records

 
Since data hashes can contain only simple object values, they can store your raw data but can’t really implement any business logic.  For example, you will often want to enforce constraints of changes, model relationships between objects, and compute additional properties.
 
Rather than returning simple data hashes, the store instead will wrap your data hashes in special model objects called records.  Records are subclasses of SC.Record that you define in your application.  This is where you can add your own business logic and define your model relationships.
 
By default, records simply pass through their properties to their underlying data hash.  When you get or set a property on a record that you haven't defined otherwise, it will get or set the same property on the data hash itself.  You can define additional computed properties on your SC.Record subclass, or use the built in attribute helpers to provide mode intelligent mapping of data hashes to your internal data model.   
 
For example, the attribute helper SC.Record.toOne() will tell SproutCore that a particular property is actually a reference to another record in your data model.  While your data hash may contain a simple string for this property, when you get() that property, the attribute helper will lookup the corresponding record in the store and return it in instead.
 
The code example below shows you what a typical record definition might look like in your application.  We’ll cover the specifics of records in more detail in a later section.
 
AddressBook.Contact = SC.Record.extend({
  
  // firstName and lastName are strings. 
  // These attribute helpers will always force them
  // to be strings.
 
  firstName: SC.Record.attr(String),
  lastName:  SC.Record.attr(String),
 
  // every contact belongs to a group.  
  // This attribute helper will map a group id to
  // a real Group record (another model in the app)
 
  group: SC.Record.toOne(‘AddressBook.Group’)
});
 
 Usually when you are working in your application, you will spend most of your time working with record instances.  This will give you a high-level view of your model without having to worry about the primitive data hashes underneath.
 
The attribute helpers defined with the DataStore can convert all primitive types (Strings, Numbers, and Booleans) as well as dates.  There are also helpers to model one-to-one, one-to-many, and many-to-many relationships.
 

Queries and RecordArrays

 
Individual records are one thing, but what about when you need to find a collection of records?  The DataStore has some features to help you here as well. 
 
Anytime you want to find a group of records, you will need to construct a query object.  A query object (derived from the SC.Query class) contains all of the parameters you want to use to scope your search, such as the types of records you want to select and other factors.  To make query objects easier to build, the DataStore includes support for a simple query language that is similar to SQL which you can use to define your query parameters.
Usually you only need to do a limited set of queries in your application.  It is often best to build these query objects up front or at least to build them only one time to avoid the store having to recompute the results over and over.  
 
When you are ready to get the records for a query, you simply call the find() method on the store, passing the query object.  The return value will be a special type of object called a record array.  A record array (derived from the SC.RecordArray) implements the SC.Array mixin, which means you can use it to drive ArrayControllers or CollectionViews.  
 
The difference is that the record array is specially designed to avoid actually creating record objects until you actually ask for them.  This means that you can create a query that might return an array of say 1,000 records, but you won’t actually pay the cost of creating – or “materializing” – these records until you actually request them.
 

Local vs. Remote Queries

 
Most of the time you will create queries that work on your local, in-memory store.  For these types of queries, the record array that is returned will update automatically whenever any record in your store is changed.  Queries that work on your in-memory store and update automatically are called local queries.
 
Sometimes, however, you can’t get the records you need simply by searching your in-memory store.  For example, you might be loading a very large number of records and need to fetch them in chunks from the server instead of all at once.  Even for a small number of records, you may simply need to depend on the server to provide your records in order.
 
In these special cases, you can create remote queries.  Remote queries will return a record array, just like a local query.  However, this time the contents of this record array will be determined by results returned from your server instead of inspecting your in-memory store.  Note that the records you actually retrieve from this array will still be held in your store; just their order and membership in the record array will be determined by the server. 
 

Data Sources

 
All of these pieces we’ve talked about so far work on data held in your in-memory store.  How does the data get there in the first place?
 
Every store is connected to an object that manages the connection with your server called the data source.  Data sources implement a special API that will be called by the store whenever it needs to retrieve records, commit changes to records, or fetch query results.  Usually you will implement this data source yourself, though there are a growing number of pre-built data sources that you can use directly or extend to meet your own needs.
 
Data sources are based conceptually on the Rack API used to chain together small web server apps in Ruby.  As in Rack, you can actually chain multiple data sources together if you need to talk to multiple backends using a cascading data source.  This data source will simply forward requests from the server one at a time to multiple other data sources until one or more of them indicates that it handled the request.
 
Since data sources contain all of the business logic related to working with your server, such as how you handle failures, conflicts, pushes or polling, etc, this is often the part of the DataStore framework that you will customize the most.  It is often the part that is the most difficult to write.  The good news is, if you construct this code properly, once it is done, you will rarely need to revisit this code.  You can then spend most of your time thinking about the rest of your app.
 

A Word About Lazy Loading and Server Performance

 
The DataStore framework is designed to work in a cloud environment where often your data will not be available right away locally.  For this reason, all store methods are designed to return records and collections of records (called record arrays) immediately; even if the data for the records is not yet available.  Once the server has actually loaded the data asynchronously, these records will automatically populate with their new values.
 
If you make proper use of bindings and computed properties in your application, the asynchronous nature of your data model will often be hidden from you.  You simply retrieve the record or record array that you are interested in and then plug it into the proper controller.  When the data arrives from the server, your controllers and UI will update automatically.  
 
Even though you will not often need to work with it explicitly, it is still important to always keep in mind the asynchronous nature of your code.  Due to the overall latency of the internet, usually any request that has to go all the way back to your server is going to take at least 250msec to return, even if your server returns data instantly.  
 
Once you take into account the typical response time for a server to go back to the database and other delays, it is usually best to design your app to expect a total 500msec delay every time you try to fetch data from the server.  This is long enough for a typical user to notice the delay so you need to plan for that in your UI.
 
On the other hand, since SproutCore apps are often very responsive, anytime you need to wait on a server round trip for some data will be far more noticeable to users than in a typical web app.  For this reason, it is imperative that you design your server to return data to the browser as fast as possible.  Given that you already pay a 250msec penalty just to go back to your server, even a 100msec server response time will make your app feel noticeably slower.
 
So keeping with the general theme of the internet to be liberal in what you receive and strict in what you send, design your SproutCore apps assuming you will have long delays for every server request but design your server as if your client required immediate response.  
 

Moving On

Learn how to Define Your Models »

or return to the DataStore Programming Guide Home »

 

Comments (2)

Levi McCallum said

at 8:19 am on Sep 6, 2009

Thank you for adding this page. Extremely helpful!

tymofi said

at 8:15 am on Sep 7, 2009

Thank you, thank you, thank you! Can't wait for the rest.

You don't have permission to comment on this page.