Swipely offers local merchants a simple way to accept payments, understand customers, and grow revenue. Swipely crunches Big Data behind every credit and debit card transaction to bring merchants powerful, easy-to-use analytics, coupled with online marketing and loyalty tools. For example, Swipely can identify a restaurant’s 25 most valuable customers, correlate Facebook and social media marketing with sales at a boutique, or even determine the impact of weather on a spa’s business. Today merchants use Swipely to manage 1 billion dollars in sales and to deepen relationships with more than 4 million customers.
Founded in 2009 by Angus Davis and named to the Forbes 100 list of “America’s Most Promising Companies,” the Providence, RI-based company includes veterans of Tellme, Microsoft, LivingSocial, and SeamlessWeb, and is backed by leading Silicon Valley investors including Shasta Ventures, Index Ventures, Greylock Partners, and First Round Capital.
Agility is key for startups like Swipely. Specifically, Swipely engineers must be able to build their latest product concepts with minimal effort, and then quickly change up to larger and more scalable infrastructure once business traction accelerates. The company began using Amazon Web Services (AWS) in 2009 to take advantage of the low startup costs, room for growth, and features of the AWS Cloud. Swipely used Amazon Elastic Compute Cloud (Amazon EC2) to provision and manage instances, defining new application-level services as they tested different product concepts. After a three-year journey to find its product-market fit, Swipely’s original lean service-oriented architecture had served its purpose, and needed to evolve to enable the fluid, flexible processing of data from multiple sources.
The company’s existing architecture wasn’t optimized for the flexible data processing that its new business demanded. “Our old architecture handled our data in a rigid, fixed way that supported the original feature set,” says Anthony Accardi, Head of Engineering. “So when we wanted to take advantage of new insights gleaned from a new data source, we would have to go back to the drawing board. It was laborious and costly.”
Swipely’s credit card processing complicates its architectural requirements, as its data security must comply with the strictest standards set forth by the Payment Card Industry (PCI DSS).
To address its flexible data processing needs, Swipely considered building its own declarative data processing pipeline to manage MapReduce clusters on Amazon EC2. During this period of deliberation, Amazon Web Services (AWS) launched AWS Data Pipeline. “AWS addressed our major pain points,” says Bright Fulton, Lead Engineering Operations. “Between the features of its new Data Pipeline service and its focus on data security, we were very comfortable letting go of our initial instinct to build our own pipeline.”
AWS Data Pipeline, shown below in the Swipely architecture, coordinates data processing and transformation resources that take facts about user actions and payments, and build analytics documents that can be efficiently queried to render on a web page or view on a mobile device.
Figure 1: Swipely Data Pipeline Architecture Diagram
AWS Data Pipeline serves an integral role in Swipely’s new data processing architecture, coordinating the processing and transformation of data between different compute and storage services. The company amasses all user actions, payment events, and external data inputs as facts in Amazon Relational Database Service (Amazon RDS) instances. Swipely engineers define the data transformations that map these raw facts to sophisticated analytics that can be efficiently queried to render on a web page or view on a mobile device. The resulting analytics documents are indexed and stored in another Amazon RDS instance.
AWS Data Pipeline rebuilds the analytics documents from the facts every night, by creating Amazon Elastic Map Reduce (Amazon EMR) clusters, executing activities on the clusters according to Swipely’s data transformation definitions, and then shutting the clusters down once the activities complete. Swipely only incurs resource usage costs for the duration required to complete the activities. The data pipeline architecture allows Swipely to mash up historical payments data with social media and user actions, providing its clients with integrated, insightful reports on demand. Moreover, Swipely can update its analytics to enable new reports and features by simply updating the data transformation definitions, as AWS Data Pipeline will rebuild all the historical data dependencies overnight.
“Customers can now see how their key business metrics change over a month, or day by day, or hourly,” says Accardi. “We run all that data through AWS to determine where the soft spots are and recommend strategies to drive more sales during those times.”
In addition to AWS Data Pipeline, Amazon EC2, Amazon EMR, and Amazon RDS, Swipely now uses Amazon Simple Storage Service (Amazon S3), Amazon Virtual Private Cloud (Amazon VPC), Elastic Load Balancing, and Amazon Route 53. Swipely currently stores and processes approximately 1 TB of data on S3.
Using AWS Data Pipeline, Swipely creates complex data processing flows that are fault-tolerant, repeatable, and highly available, reducing complex feature development time from weeks to hours. “Say a customer wants a completely different take on their data,” Accardi says. “In the old architecture, we’d have to start at square one to build that. Now we just change our existing AWS Data Pipeline definitions, and we’ll have our entire data set rebuilt by the next day. Instead of writing lots of complicated code, we store and transform lots of simple data.”
The pay-as-you-go pricing offered by AWS helps Swipely control costs. “AWS Data Pipeline schedules enough on-demand instances to crunch all our data every night, so we only pay for resources when we need them,” Fulton says. “AWS manages resource allocation and inter-task dependencies so we don’t have to. It’s pretty darn efficient.”
Swipely leverages AWS to meet stringent PCI security standards. “With the option to encrypt data at rest and PCI certification, Amazon S3 is tailor-made for meeting PCI requirements,” Fulton says. “We flew through rigorous certification because we put simple mechanisms in place to ensure security retention policies on Amazon S3. I don’t see many other cloud providers making security as easy as AWS.”
AWS enables Swipely to create dynamic reports and make data-driven recommendations for clients in seconds, no matter what source the data comes from. “We’re able to manage massive amounts of data easily, because of AWS,” Accardi says. “The impact on our company is huge.”
“Using AWS, we make major application-level changes overnight—within hours instead of weeks,” Accardi concludes. “It’s very powerful. We can concentrate more on our product and our market, and less on infrastructure management.”
To learn more about how AWS can help your data needs, visit our Big Data details page: http://aws.amazon.com/big-data/.
For more information about data pipelines, see the AWS Data Pipeline page: http://aws.amazon.com/datapipeline/.
