Greg Rokita, Director, Senior Technical Architect at Edmunds, gave a talk last weekend at SoCal Code Camp about the best practices of designing a publishing system for high volume sites
In his talk, Greg discussed:
The talk generated a lot of excitement about our publishing system. We would love to hear what you think!
Paddy Hannon, VP of Enterprise Software and Data Architecture, talks about Oracle Coherence in a series of Brown Bags at Edmunds, Inc.
In this video, Paddy covers the following:
- Coherence Data Distribution Models
We have been using Oracle Coherence for a little over 2 years now. All things considered it has served us well - specifically in regard to keeping our data backward compatible. What I mean by "backwards compatability" is the ability to modify data points on existing domain objects without causing applications which don't yet require the new data points to be upgraded.
Coherence has a technology called "POF" (portable-object-format) which provides this capability. This article only dives into one of the many uses of POF, related to "backwards compatibility", as there are many other uses for POF that are not explored in this article. Also, before you continue reading please check out Paddy's video on Coherence if you aren't very familiar with what this technology does.
First off objects stored in Coherence must be serializable by one of the following formats:
Objects must be serializable because Coherence is a "distributed" object caching system, meaning that even if one JVM creates the object it will most likely end up being serialized across the network to another JVM responsible for caching it. Since our objects represent our "data", you need to make sure that you store your objects in an extensible manner so that data modifications can be made without causing a ripple effect of application deployments. Here at edmunds we have over 50+ different deployable applications (WAR, standard java app, etc) that use various distributed caches, so we cannot afford to re-deploy all of our applications if we are simply refactoring objects.
Taking the above into consideration when choosing a serialization format, Java serialization does have the ability to allow classes to be "versioned" to some extent by using the "serialVersionUID" (following these guidelines), it still is slow and produces a rather large object. Coherence has two additional forms of object serialization to help aid both performance and extensibility:
While POF has lots great features, the one feature we find very useful is that it allows object data to be "versioned" very easily. This has proven to be very useful to keep our data backward compatible so that no application is broken if another is upgraded or updated.
Take the example below with call "Person":
public class Person extends AbstractEvolvable implements
EvolvablePortableObject {
private String firstName;
private String lastName;
private static final int POF_DATA_VERSION = 1;
private static final int FIRST_NAME_POF_INDEX = 1;
private static final int LAST_NAME_POF_INDEX = 2;
/**
* {@inheritDoc}
* Coherence uses this method before calling writeExternal().
*/
public int getImplVersion() {
return POF_DATA_VERSION;
}
public void readExternal(PofReader pofReader) throws IOException {
firstName = pofReader.readString(FIRST_NAME_POF_INDEX);
lastName = pofReader.readString(LAST_NAME_POF_INDEX);
this.setFutureData(pofReader.readRemainder());
}
public void writeExternal(PofWriter pofWriter) throws IOException {
// POF_DATA_VERSION is written automatically by Coherence before
// calling this method.
pofWriter.writeString(FIRST_NAME_POF_INDEX, firstName);
pofWriter.writeString(LAST_NAME_POF_INDEX, lastName);
if(this.getFutureData() != null) {
pofWriter.writeRemainder(this.getFutureData());
}
}
}"Person" is using POF (POF_DATA_VERSION = "1") and has two properties: firstName and lastName. Now let's assume that the serialized "Person" object is now live in production and being used by several applications. Now let's say that gender becomes a critical property of the "Person" object and we need to add it. Now the call will look like:
public class Person extends AbstractEvolvable implements
EvolvablePortableObject {
private String firstName;
private String lastName;
// New POF field
private String gender;
// Up the data version to 2
private static final int POF_DATA_VERSION = 2;
private static final int FIRST_NAME_POF_INDEX = 1;
private static final int LAST_NAME_POF_INDEX = 2;
// New POF index for gender.
private static final int GENDER_POF_INDEX = 3;
public int getImplVersion() {
return POF_DATA_VERSION;
}
public void readExternal(PofReader pofReader) throws IOException {
firstName = pofReader.readString(FIRST_NAME_POF_INDEX);
lastName = pofReader.readString(LAST_NAME_POF_INDEX);
// Only attempt to read gender if your POF data version
// is at least equal to or greater than the version of the data
// being read in via PofReader.
if(pofReader.getVersionId() >= 2) {
streetAddress = pofReader.readString(GENDER_POF_INDEX)
}
this.setFutureData(pofReader.readRemainder());
}
public void writeExternal(PofWriter pofWriter) throws IOException {
pofWriter.writeString(FIRST_NAME_POF_INDEX, firstName);
pofWriter.writeString(LAST_NAME_POF_INDEX, lastName);
// No logic here as we are just adding a field and our data is
// always only written to the cache in one place - the JMS
// listener.
pofWriter.writeString(gender_POF_INDEX, gender);
if(this.getFutureData() != null) {
pofWriter.writeRemainder(this.getFutureData());
}
}
}The updates above are:
Note that we hardcoded the version (2) that is required to read in the "gender" field. We did this so that when future modifications are made it is easy to determine what version a particular data point is compatible with.
Another thing to point out is that you always should increment when adding additional fields to a "POF'able" object as there may be another version of that same object that is writing a totally different type of field at that index which would cause ClassCastExceptions, etc.
Now both version 1 and 2 of the Person object can co-exist in production at the same time in our data grid and the applications that currently use version 1 will remain intact while we can create or update other applications to take advantage of the data available in version 2. In effect, versioning allows for peace of mind when it comes to deploying updated data objects or refactoring our data to accommodate new business requirements.
Hopefully this article showcases one of the ways in which POF can be useful as your serialization format. I say this because there are many other reasons to consider POF as your serialization format other than backwards compatibility alone.
The actual coding of software is really just one small piece in a much the larger fabric of delivering bona fide functional value to users. Here at Edmunds, on the eve of the full site launch of our redesigned site (two years in the making), we find ourselves in the new.
New infrastructure, new processes, new tools, new ways of working with each other and new ways of thinking about ourselves, i.e., how do we as technologists (or for that matter as individuals) provide optimal value in an organization that has organically, seemingly overnight, evolved to place where nearly everything that has successfully brought us where we're at is being turned on its head? The roles that we've played and hung our collective hats on are, for the first time, being significantly redefined in the pursuit of innovation: to break out of the practice of doing relatively small, incremental improvements and into producing something truly different.
The general ethos of the Edmunds team has come to re-conceptualize "software development" to be the broader end-to-end process cycle of delivering value to the user: from idea creation (what to build & why) to production deployment (what, when, who and how often). What this holistic view ultimately means is that Software Development = Product Development. So as the organization starts a new chapter with the adoption of Design Thinking, it feels surprisingly comfortable alongside concepts established (or at least in the process of being established) such as Agile software development, Dev Ops, and Test-driven development (TDD), Scrum, and Kanban swarming. In fact, in a way, it's feels like the natural evolution of these practices. It feels comfortable because unbeknown to ourselves, we've evolved.
So I find myself asking, if software has no value until it's in the hands of real people (users), then what are the things that stand between the idea and effective instantiation of that idea? The answer to the question, and following the question to its natural conclusion (what should be done about it?), is a set of even newer infrastructure and newer process and newer tools. Moreover, it involves yet further redefinition of roles that have to be first broken before they are truly remade. Reevaluating the tasks and responsibilities of roles like "software engineer", "QA engineer" or "release manager" is just the start. A deeper look will naturally lead to questioning some of the principles that have historically been sacrosanct: the requisite checks & balances between Dev & QA, governance of what gets approved (or not) for a release, the concepts of what is a "release", the ceremony around actually pushing functionality to production. Many of these conventions exist to stabilize an inherently dysfunctional system. A system where developers hack out code with little regard for building quality into their code; a system inflexible and so prone to errors that it must be tightly and painfully gated to ensure adequate quality.
But what happens when an organization has sufficiently evolved beyond this?
Stay tuned and find out.
Edmunds is on the news a lot. People such as CEO Jeremy Anwyl, Senior Analyst Jessica Caldwell, Senior Consumer Advice Editor Philip Reed and the rest of our expert editorial crew get a lot of air time.
You'd think that these people were on airplanes all the time flying around the country to various studios. But, they're not.
Actually, the reality of the situation is a lot more interesting. Here at Edmunds we actually have a broadcast studio. It's not big. It's a room about 10 ft. by 10 ft.. You could mistake it for someone's office. But, just about every time you see someone from Edmunds on TV, if he or she is not directly at a table with an interviewer, we're broadcasting from this little room. On your television screen it seems as if we have facilities on par with CNN.
So how does all this work?
We use a product named, ReadyCam by VideoLink. Our ReadyCam installation is turnkey. We paid VideoLink to come in and setup the studio for us.
They put in the lighting, camera and earphone hook up. The complete broadcast is handled by VideoLink remotely. They manage our transmission, acting as a virtual cameraman, lighting and sound technician.
Say, MSNBC wants to do an interview with one of our Industry Analysts. MSNBC will coordinate with VideoLink and Edmunds to determine an air time. Prior to the broadcast one of our media technicians goes into the studio and turns on the room equipment. Once the equipment is powered up, VideoLink takes over.
Our analyst has to nothing more than sit at the studio desk and be an expert. VideoLink hands off the video feed to MSNBC and acts as cameraman for the MSNBC director running the entire broadcast.
The analyst interacts with the broadcaster via the remote controlled earphone that's part of the ReadyCam package. If something goes wrong in the studio, VideoLink takes care of the problem.
Using ReadyCam, we're able to create broadcast quality content without having to incur the cost of staffing and maintaining a broadcast quality studio.
It's pretty cool.
I wrote a short post about our data grid topology on my personal blog a few days ago. However, I feel that the topic deserves more conversation, so I'm going to cover it in a bit more detail here.
When we first started out, we used a RDBMs as the data repository to power our site. We had a very classic 3-tier architecture with big servers sitting behind the site that powered a collection of relational databases. As part of our development cycle, we moved code from a team's development servers to a shared development integration environment. From there, we deployed to a QA environment, then to a Staging environment and finally to Production.
At each stage the server configurations start to look more and more like Production. I won't get into how painful the deployment process has been in the past because we have made large improvements in automating our deployment processes. The point I want to stress here is that each environment had a full stack. At any given time there were at least four central database server farms that needed to be kept up-to-date with the latest data. In addition, there were times when a schema change was making its way through the stack, thus making the task of keeping data fresh even more problematic.
With our new architecture, we have moved away from a relational database. Now we are leveraging a Coherence data grid and Solr search servers. However, we have kept our original topology of having a full stack per environment.
But I still question myself: Given a modular, service-orientated architecture, does keeping a full stack in each of our many environments make sense? My hypothesis is that a full stack does not make sense.
With some of our other systems we have moved away from having a full stack per environment. For example, our publishing system is deployed as a shared resource. All of our environments plug into this shared environment. The publishing system still supports Development, QA/TEST, and Production environments. Changes to the publishing system still go through dev and test environments prior to being deployed into production, however, that stack is only used for pushing changes to the publishing system and only interact with pre-release environments to test actual changes to the publishing system. At any given point, all environments use the production version of the publishing system. My thinking is that we could take the same approach to our data storage systems. That is, both Solr and Coherence data services could be moved through a track where all environments would plug directly into the production version of the data service.
The advantage of working off of the production data service is that all environments would be in sync with data. Also, developers would be able to test changes more easily and ensure that their new code will work with what is in production. Such a deployment topology will allow us more visibility and control. We'll know the versions of our data services that are being used. Also, this new deployment topology will provide a streamlined mechanism for delivering changes to our data services. Since developers will be managing a shared resource with multiple clients, the data service developers will need to consider backwards compatibility while developing their code.
The disadvantage of using a deployment topology that works off of production data is that bad code in development can affect our production web site. This is a pretty big deal for Edmunds. Our entire revenue stream is derived from the web. Perhaps, what we need to do is use two production grids--one for internal/pre-Production use and one for the Production website?
What are your thoughts? Have any of you considered alternative deployment strategies for your data services? If so, what have you tried? I'd love to hear from anyone out there that has ideas, comments, suggestions.
Aloha,
Paddy Hannon
This Saturday and Sunday, October 23-24, some of us will be talking at SoCal Code Camp at USC.
On Saturday, Greg Rokita will be giving a talk on Best Practices for Architecting High Volume, High Performance Publishing for Data Intensive Web Site. During the same time slot, I will be giving a talk on Mitigating Advertisement Impact on Page Performance. On Sunday, Bob Reselman will give a talk on The Most Important Technology @ SoCal Code Camp.
We are looking forward to it. Stop by and say hello if you can. It would be great to hang out with fellow geeks.
Edmunds is on the news a lot. People such as CEO Jeremy Anwyl, Senior Analyst Jessica Caldwell, Senior Consumer Advice Editor Philip Reed and the rest of our expert editorial crew get a lot of air time.
You'd think that these people were on airplanes all the time flying around the country to various studios. But, they're not.
Actually, the reality of the situation is a lot more interesting. Here at Edmunds we actually have a broadcast studio. It's not big. It's a room about 10 ft. by 10 ft.. You could mistake it for someone's office. But, just about every time you see someone from Edmunds on TV, if he or she is not directly at a table with an interviewer, we're broadcasting from this little room. On your television screen it seems as if we have facilities on par with CNN.
So how does all this work?
We use a product named, ReadyCam by VideoLink. Our ReadyCam installation is turnkey. We paid VideoLink to come in and setup the studio for us.
They put in the lighting, camera and earphone hook up. The complete broadcast is handled by VideoLink remotely. They manage our transmission, acting as a virtual cameraman, lighting and sound technician.
Say, MSNBC wants to do an interview with one of our Industry Analysts. MSNBC will coordinate with VideoLink and Edmunds to determine an air time. Prior to the broadcast one of our media technicians goes into the studio and turns on the room equipment. Once the equipment is powered up, VideoLink takes over.
Our analyst has to nothing more than sit at the studio desk and be an expert. VideoLink hands off the video feed to MSNBC and acts as cameraman for the MSNBC director running the entire broadcast.
The analyst interacts with the broadcaster via the remote controlled earphone that's part of the ReadyCam package. If something goes wrong in the studio, VideoLink takes care of the problem.
Using ReadyCam, we're able to create broadcast quality content without having to incur the cost of staffing and maintaining a broadcast quality studio.
It's pretty cool.
I wrote a short post about our data grid topology on my personal blog a few days ago. However, I feel that the topic deserves more conversation, so I'm going to cover it in a bit more detail here.
When we first started out, we used a RDBMs as the data repository to power our site. We had a very classic 3-tier architecture with big servers sitting behind the site that powered a collection of relational databases. As part of our development cycle, we moved code from a team's development servers to a shared development integration environment. From there, we deployed to a QA environment, then to a Staging environment and finally to Production.
At each stage the server configurations start to look more and more like Production. I won't get into how painful the deployment process has been in the past because we have made large improvements in automating our deployment processes. The point I want to stress here is that each environment had a full stack. At any given time there were at least four central database server farms that needed to be kept up-to-date with the latest data. In addition, there were times when a schema change was making its way through the stack, thus making the task of keeping data fresh even more problematic.
With our new architecture, we have moved away from a relational database. Now we are leveraging a Coherence data grid and Solr search servers. However, we have kept our original topology of having a full stack per environment.
But I still question myself: Given a modular, service-orientated architecture, does keeping a full stack in each of our many environments make sense? My hypothesis is that a full stack does not make sense.
With some of our other systems we have moved away from having a full stack per environment. For example, our publishing system is deployed as a shared resource. All of our environments plug into this shared environment. The publishing system still supports Development, QA/TEST, and Production environments. Changes to the publishing system still go through dev and test environments prior to being deployed into production, however, that stack is only used for pushing changes to the publishing system and only interact with pre-release environments to test actual changes to the publishing system. At any given point, all environments use the production version of the publishing system. My thinking is that we could take the same approach to our data storage systems. That is, both Solr and Coherence data services could be moved through a track where all environments would plug directly into the production version of the data service.
The advantage of working off of the production data service is that all environments would be in sync with data. Also, developers would be able to test changes more easily and ensure that their new code will work with what is in production. Such a deployment topology will allow us more visibility and control. We'll know the versions of our data services that are being used. Also, this new deployment topology will provide a streamlined mechanism for delivering changes to our data services. Since developers will be managing a shared resource with multiple clients, the data service developers will need to consider backwards compatibility while developing their code.
The disadvantage of using a deployment topology that works off of production data is that bad code in development can affect our production web site. This is a pretty big deal for Edmunds. Our entire revenue stream is derived from the web. Perhaps, what we need to do is use two production grids--one for internal/pre-Production use and one for the Production website?
What are your thoughts? Have any of you considered alternative deployment strategies for your data services? If so, what have you tried? I'd love to hear from anyone out there that has ideas, comments, suggestions.
Aloha,
Paddy Hannon
This Saturday and Sunday, October 23-24, some of us will be talking at SoCal Code Camp at USC.
On Saturday, Greg Rokita will be giving a talk on Best Practices for Architecting High Volume, High Performance Publishing for Data Intensive Web Site. During the same time slot, I will be giving a talk on Mitigating Advertisement Impact on Page Performance. On Sunday, Bob Reselman will give a talk on The Most Important Technology @ SoCal Code Camp.
We are looking forward to it. Stop by and say hello if you can. It would be great to hang out with fellow geeks.
Paddy Hannon, VP of Enterprise Software and Data Architecture, talks about Oracle Coherence in a series of Brown Bags at Edmunds, Inc.
In this video, Paddy covers the following:- Coherence as a distributed hashmap
- Serialization methods in Coherence
- Coherence Portable Object Format, or POF.
A couple of books have been circulating around the office for the past few weeks. Both subject matters are directly related to our objective to continually promote and support a culture of innovation and creativity. The first book, Open Leadership by Charlene Li (Kindle|Print), discusses how to be a leader in a time where everyone is endlessly connected and where opinions and biases can make or break a business. The other book, Where Good Ideas Come From by Steven Johnson (Kindle|Print), presents a brief history of innovation through amusing anecdotes and provides a set of tools that help spot and nurture good ideas.
The two books might seem to cover unrelated subjects at a first glance, yet they are connected with common themes; Most notably, the concepts of The Adjacent Possible and Liquid Networks. These two themes have also been covered by Malcolm Gladwell in both Blink (Kindle|Print) and Outliers (Kindle|Print). Just about any business book that talks about innovation and creativity covers these two themes to a certain extent.
Steven Johnson writes that, "what the adjacent possible tells us is that at any moment the world is capable of extraordinary change, but only certain changes can happen." At the heart of the Adjacent Possible theory is that good ideas (or anything for that matter) will come to exist and thrive if and only if all the elements needed for its survival already exist and the proper connections between them are made.
That's what Charlene Li does for us in her Open Leadership book. She identifies the elements needed and the connections required to accomplish open leadership. She provides the parts and we're responsible for building the machine. The same is true for "thin-slicing" that Malcolm Gladwell talks about in Blink. Our ability to make quick decisions based on very limited information is a direct result of many years of cerebral and emotional growth that couldn't have been possible had the right elements not been available or the proper connections not been made. Come to think of it, the Adjacent Possible is the basis of every evolution be it natural or man-made.
The other theme is Liquid Networks, which references environments that promote what Steven Johnson calls, "information spillover." He uses MIT's Building 20 and Microsoft's Building 99 as examples of fluidity in the office space. Charlene Li's call for open leadership inherently promotes liquid networks. In Malcolm Gladwell's Outliers, the environments in which his subjects thrived were all liquid networks.
At Edmunds, we realize that there are a finite set of permutations of space, talent, and process that once implemented could create a fertile environment where extraordinary ideas become a reality. We have been exploring the "edges of possibility" for a while now and I'd like to think that we've made considerable progress. We have adopted Agile rather successfully for over three years now and we're currently experimenting with Design Thinking and seating arrangements to promote "information spillover" and hone our skills of identifying the parts needed to make our ideas adjacent possible.
This very blog and our soon-to-be-released APIs, products and tools are also part of our effort to expand our circle of influence for a broader information spillover. We believe that the community will identify and build new parts that will make bigger and grander ideas enter the realm of adjacent possible.
Above all, we believe highly in our talent and we are always looking for passionate individuals to join our ranks.
Exciting times do lie ahead :-)
One of things that we've been successful at hear at Edmunds is writing technical documentation that is clear and useful. It's taken a while. But now we're getting pretty good at it.
An important first step that we took toward improving the quality of our technical documentation was giving a lot of attention to eliminating the use of indeterminate pronouns.
What is an indeterminate pronoun? Consider the following sentence shown in Listing 1.
Listing 1: A sentence using an indeterminate pronoun
When building a REST service using a Cloud based application server, you'll do best
to create a prototype of it first.In the sentence shown above in Listing 1, and as diagrammed in Figure 1 below, please take note of the second to last word, it. Does "it" reference to the REST service or to the Cloud based application server? You really don't know for sure.
Figure 1: An indeterminate pronoun is one in which the noun being referenced is ambiguous
Now, if the sentence was written as such:
When building a REST service using a Cloud based application server, you'll do best
to create a prototype of the REST service first....there is no ambiguity.
As I mentioned earlier, the use of indeterminate pronouns is the curse of technical documentation. If you're confronted with limited resources and personnel, the easiest way to clean up an organizations technical documentation is to address the use of indeterminate pronouns.
Here at Edmunds we use the wiki product, Confluence. One of the nice things about Confluence is that it allows us to create custom user macros that are used by writers and editors to make technical writing a little easier. We've created a bunch of user macros that allows editors to mark up a document when doing a technical edit. One of these technical editing macros that we've created is the {pro} macro. ({pro} is shorthand for indeterminate pronoun.)
Listing 2 below shows you how we use the {pro} macro to let writers know that there is an indeterminate pronoun in play.
Listing 2: The {pro} macro is a customer user macro written under Confluence.
When building a REST service using a Cloud based application server, you'll do best to create a prototype of ???Indeterminate pronoun[ it ]??? first.The {pro} user macro is easy to use and highly effective. So, for all of you out there that work in a company that uses Confluence, we're posting the code for the {pro} macro so that you can add it to your technical editing toolkit. The code for the {pro} macro is displayed below in Listing 3:
Listing 3: The code for the {pro} Confluence user macro
<span style="font-weight:bold;color:red">???Indeterminate pronoun[</span>
$body <span style="font-weight:bold;color:red">]???</span>
Our first of many more iPhone apps went live on iTunes this week. The app helps those who are in the market for a new car to research Edmunds.com's new car inventory and get the right price and True Market Value, or TMV, for a specific vehicle. This information empowers the car shopper to negotiate with the dealer on the best price for the car.
The app also allows the consumer to configure a car to his or her specifications. Once the configuration is set up, it could be emailed for record-keeping or sharing with friends.
Once a particular vehicle is selected, a "Photo Gallery" option is enabled to give you the opportunity to browse through the available vehicle images by flicking back and forth from one photo to another.
The application is also location-aware. When a vehicle is selected, a list of dealers near you are displayed in a list or on a map. Reviews on those dealers and their contact information are also provided in the app.
We are listening to your feedback. Our Mobile and Emerging Technology Team is already working on adding enhancements to the app and fixing any reported bugs. We value our customers greatly and we are always working making sure that is reflected in all our products and services.
We would love to hear from you about your experience with the app. Download it to your iPhone and rate it to let us know what you think. We're also working on more iPhone and Android apps to come your way soon. So stay tuned!
Below are some screen shots from the app:
