Facebook Meets Amazon

The Client

A world leader in the manufacturing and development of springs came to us. Now an almost billion dollar company, they evolved from a small operation to become a major player in the global manufacturing market. Looking to find new markets, some young blood in the family started looking at new sectors to build companies in. Working with industry experts in special markets, they saw potential in online shopping. They wanted to become a dominant player in the space, so they created a grand vision to create something special, a social platform centered around products. Nobody had ever done anything of this size in this space. Thus, the social search engine for sales was born.

The Problem

The site was a very complex ecosystem. The social platform revolved around the products platform. The product's platform needed to have everything a user could want: hundreds of millions of products from hundreds of stores. More importantly, the pricing needed to be constantly updated. Even the slightest price percentage change needed to be tracked and pushed to users because that could make the difference in engagement. The utility of the products platform is what energized the social platform. Without accurate pricing, there was little that could make it go viral organically, which is needed for community plays in markets nowadays.

We were brought in after several engineering and marketing firms failed to deliver for the client. The firms were large, highly recommended, award-winning firms but they still could not get the job done. They promised them the world, assuring them they could handle a project of their size, but it was clear they couldn't. With millions of dollars invested, the future looked bleak without someone who could execute.

Design

Despite a major investment of time and energy with specialized design firms, the site had a very basic design. It was not converting around 2% of the time. People were not taking the actions they desired like signing up or converting to sale. Bounce rates were extremely high at over 80%.

Speed

With less than 100,000 items, the site speed was very slow taking around 15-30 seconds to load a single page with barely any concurrent users. Despite having beefy servers, they could not handle the data they already had. However, they would need around 10,000 times more data to come close to enough products.

Complexity

Their goal was to incorporate diverse data from many different kinds of sources. But the firms they hired were struggling to handle data from just a single source. They had not even scratched the surface of complexity and were showing their limits.

Technical Advisors But No Progress

After the first few failed contracts, they tried to make a change. Being non-technical, the executive team brought in seasoned engineering veterans to serve as technical advisors to screen and manage the work being done for them. Despite this, progress was not there.

Investor Confidence

Even if a startup's failure is not the fault of the founders, the investor still holds them accountable. They were suffering from a severe lack of talent to do the work they needed to be done. Investor confidence was running out. Was their vision just too big? The investors were not interested in putting in another round until they could prove things would be different.

The Solution

The problem with their project was that it was simply too complex for most engineers. Few firms could handle the technical and business requirements to pull it off in a cost-effective manner. We knew we had to have a really solid strategy going into this. We realized that the key to their success lied in the investor's confidence. Nothing brings more confidence to an investor than user engagement. We prepared a plan of action to optimize their experience to drive engagement and revenue KPIs. With these key metrics trending upward in a major way, it would be much easier to get investors on board with future rounds.

Design

The original design is very cluttered and messy with too many sections crammed into small spaces. It was visually confusing and overwhelming for new users. A fresh, modern design was created to replaced the dated original design. We wanted a look that was unique but also one that more closely matched online shopper expectations. The desired aesthetic was a clean, minimalist look that mirrored higher end brands. These changes boosted engagement tremendously. The conversion rate surged to nearly 40%.

Speed

With the scale they wanted to get to, speed was a major issue. They needed a system that worked for them now but would also work in the future. With limited funds in the investment round, they could not afford to build a temporary system now and then another system later. They needed something built right the first time.

We built an automated parallel distributed system to handle all data processing through their system on both the backend and frontend. Everything was being done synchronously so we broke down synchronous processes into parallel chunks that could be performed atomically with high reliability. This lead to single-threaded speedups of over 1,000x running on a single machine compared to their old systems. When spinning up server clusters, the multi-threaded improvements were well over 10,000,000x. This system was infinitely scalable with server costs scaling much slower than the computing power. To optimize costs, servers were scaled up and down as the workload changed to avoid having too many wasted idle server resources.

Complexity

With so much content coming from so many sources, we knew certain key features could make or break the project's performance. The biggest of these was the parsing system. The parsing system transforms content from any source into a unified data model for consistent, predictable use across the system. This completely removed parsing-related complexity out of all code besides the parser module. This greatly simplified the core code.

We parsed extremely diverse data from different platforms with different methods (ex: API, web scraping, pulling feeds), formats (ex: XML, JSON, CSV, TSV, HTML, SOAP), protocols (ex: HTTP, FTP) and authorization schemes (ex: OAuth, HTTP, keys). This made handling data predictable.

With parsing out of the way, we created a modular infrastructure for the many different features the system would need. This would allow future engineers to do major development work in parallel. The final system would continuously find, parse, import and update items from many sources. They were indexed into an Amazon-like search engine. The total item count in total neared 1 billion items. These items were served in fractions of a second on search frontend.

The Results

10,000x

Scaled To More Content

50x

Search Response Speedup

1,000x

Single Processing Speedup

10 Million x

Parallel Processing Speedup

2,000%

Conversion Rate Improvement

108x

Data Sources Supported

The Lesson

Not All Technical Talent Is Equal

With so many companies bidding for your work, having a technical advisor as an advisor to sift through them is critical. Technical advisors are great as double insurance. But as they learned, just having a technical advisor does not mean everything will magically work. They introduced technical advisors, but they were still not seeing progress. They could see more problems, but it wasn't moving them forward.

That is because not all technical talent is equal. There are many dimensions to technical work. In engineering, for example, there is frontend, backend, dev ops, web, mobile, and much more. Having technical talent that specializes in only one of these areas means they can't serve that advisory for the entire product cycle which includes all the areas. You need someone with high-level experience in all the areas which is rare outside of consulting firms. That is the difference between consulting firms and development agencies.

Highly Recommended Does Not Mean Highly Skilled

A common misconception for clients is that if someone is highly recommended, they must be highly skilled. The truth is that the companies that get the most business do not have the best talent; they just have the best sales team. It is the same with products: the most popular product is always the one with the best marketing, not the one that is technically the best. This is a major problem when you need a skilled team for a project.

The best way to find skilled candidates is with a technical advisor. Without one, there is still an effective way to cut through the sales pitches: ask for their results. Specifically, ask for what numerical impacts they achieved. Marketing teams are great at spinning results, claiming "amazing" project results when the real change was only 5%. You can overemphasize qualitative results to seem more impressive, but numbers don't lie.