Constantly asking a data provider for updates is the current status quo for most brokers and data consumers in the real estate industry. Data producers pushing updates out to consumers only when needed could be a reality in the very near future with a straightforward new technology model.
Constellation1 has implemented RESO’s new standard for push notifications. They explained how their efforts can improve an organization’s data-gathering efficiency in a talk at the RESO 2022 Fall Conference titled “Pushing vs. Polling: The Future of MLS/Broker Data Distribution” presented by Rick Herrera, Vice President of R&D, Data Services at Constellation1, and chair of the RESO Data Consumers Subgroup. | WATCH VIDEO (27:44)
At the heart of this evolution is the acquisition of real-time data. Most real estate data is not truly real-time. The industry’s current practice of frequent polling was introduced about 20 years ago with the advent of the Real Estate Transaction Standard (RETS).
Polling means that you ask a server if anything has changed in its data set since the last time you checked. It replaced data downloading by way of manual file transfer protocol (FTP), which was a tremendous upgrade at the time. But the challenges of polling have grown after two decades in operation.
Where servers used to be polled four times an hour, there are now systems that poll every 15 seconds. If a data consumer is polling 500+ MLSs every 15 seconds, that is nearly 2.9 million actions per day. This kind of data slamming, which has become more regular in a competitive business environment, has product, operations, speed and business costs. It has spurred responses like data quotas and rate limiting from data providers, which adds additional frustration to relationships between the parties involved.
As a result of direct feedback from consumers, data providers and vendors, RCP-027 (RESO EntityEvent Resource and Replication Model) and RCP-028 (Push Replication Using Webhooks and the RESO EntityEvent Resource) were created, discussed and vetted by the RESO Transport Workgroup.
Webhooks provide a way for apps to send automated messages to other apps when an event occurs. If anything changes, the consumer gets pinged. This is commonly known to consumers as “push notifications,” which most people with modern phones are familiar with.
With an abundance of excitement, Constellation1 got right to work on building a simple API to account for RCP-027 and RCP-028. The additions were completed in a matter of hours, not days. Kevin Yao, Senior Cloud Engineer at Constellation1, explained how they went about this.
Their architecture is built upon Amazon Web Services (AWS), and they were able to add what they built without changing anything in their existing Web API system. Data comes from MLSs, gets processed through a data pipeline and gets consolidated into a single stream (including property updates, open houses, member data, etc.).
From here, they built a simple notification service as a subscription that taps into the stream for webhook events. Clients then set up their own APIs, and information is pushed to them.
The push notifications are sent to a URL by way of the subscription. Yao suggested this model to avoid what can quickly become an overwhelming amount of communications, such as email notifications.
The payload consists of identifying information so the consumer can go pick up the new data: the record’s unique ID, source, time created, sequence number and a URL with its location.
Constellation1 is already learning from its own creation. They have included a test endpoint for clients to experiment with the system before committing to it, and they have asked for direct feedback to help with advancement, improvement and increased adoption. Here are some items that are presently on their radar:
- Full payload delivery
- Additional security (e.g., message-level encryption)
- Subscription pause
- Custom retry settings
RE/MAX, RentSpree and Redfin were noted as some of the first companies to test the new system and help get it off the ground.
Data consumers and vendors want the same thing: to get current and accurate information to consumers as quickly as possible at the lowest resource cost. Mindless polling that delivers unchanged results about 80 percent of the time is inefficient and costly. Lambda functions, step functions, rights, logs, storage, memory and reporting all have hefty price tags with polling.
It is time to move beyond a methodology that was popularized when the Blackberry was the de rigeur tech. Rick at Constellation1 (rick.herrera@constellation1.com) and RESO (dev@reso.org) are ready to be your industry partner toward a more efficient future of real estate data.
